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Introduction 


Part  1  Introduction 

Abstract:  This  report  describes  impiementation  guidance  for  Version  3.0 
of  the  Software  Capability  Evaluation  (SCE)  method.  This  version  of  the 
Implementation  Guide  is  updated  to  reflect  the  new  method  and  provides 
specific  guidance  for  selecting  software  product  and  services  suppliers  in 
an  acquisition  application  (government  or  commercial)  and  provides  sug¬ 
gested  language  and  examples  of  usage. 


1 .1  About  This  Document 

Purpose  This  document  provides  software  organizations  with  guidance  for  planning 

and  implementing  Software  Capability  Evaluation  (SCE)  '"^appraisals  in 
various  software  efforts  (e.g.,  full-scale  development,  maintenance).  This 
guide  addresses  such  topics  as  incorporating  SCE  into  government 
""♦^source  selections,  "^process  monitoring,  and  the  use  of  SCE  by  con¬ 
tractors  to  determine  suitable  teaming  or  prime  contractor-subcontractor 
arrangements. 

SCE  can  help  acquisition  managers  achieve  the  following  goals: 

•  Identify  program  risk  by  evaluating  "^software  process  capability  in 
source  selection. 

•  Manage  program  risk  by  motivating  contractors  to  improve  their  soft¬ 
ware  development  processes  without  forcing  compliance  to  specific 
practices. 

This  guide  can  be  used  to  implement  the  SCE  Method  and  help  achieve 
the  goals  above.  This  guide  provides  the  information  necessary  to  orches¬ 
trate  SCE  during  the  source  selection  process.  Specifically,  this  guide 

•  Provides  guidance  on  how  to  use  the  SCE  method  as  a  tool  to  identify 
software  risk  during  a  source  selection. 

•  Provides  standardized  SCE  implementation  guidance  which  is  docu¬ 
mented,  available  for  review  and  comment,  and  periodically  modified 
as  experience  is  gained  with  its  use. 


An  arrow  ('«•■)  preceding  a  term  in  boldface  type  indicates  that  the  term  is  defined  in  the  Glossary  on  page  99. 
This  format  is  used  only  on  the  first  occurrence  of  a  glossary  term. 
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•  Provides  information  which  will  help  acquisition  organizations  deveiop 
appropriate  policies,  implementing  instructions,  and  guidelines  to  use 
SCE  in  source  selection  and  institutionalize  SCE  as  a  routine  practice. 

•  Supplements,  but  does  not  replace,  team  training  for  evaluating  the 
software  process  capability  of  contractors. 

Audience  The  primary  audiences  for  this  document  are  government  and  industry 

software  acquisition  and  software  development  managers  and  SCE  lead 
evaluators.  Two  perspectives  are  offered  in  Part  3.  The  primary  perspec¬ 
tive  is  that  of  an  SCE  "^evaluator — ^the  individual  responsible  for  carrying 
out  SCE  activities.  The  second  perspective  is  that  of  the  '"♦recipient.  A  re¬ 
cipient  is  the  organization  or  person  responsible  for  “receiving”  or  being 
subject  to  an  SCE. 

structure  This  document  is  divided  into  four  parts: 

Part  1  introduction 

Part  2  Overview  of  SCE  Implementation  Guidance 
Part  3  Activity  Implementation  Guidance 
Appendices 


1 .2  Background  and  Context 

Software  Capability  Evaluation  (SCE)  is  a  method  for  evaluating  the 
software  process  of  an  organization  to  gain  insight  into  its  software  devel¬ 
opment  capability.  This  insight  can  be  a  valuable  input  to  process  improve¬ 
ment  activities. 

Hence,  the  SCE  Method  helps  evaluate  the  software  process  capability  of 
a  software  '•♦^development  organization  (an  organization  that  develops 
and/or  maintains  software  products).  Software  process  capability  refers  to 
the  range  of  expected  results  that  can  be  achieved  by  following  a  process. 

The  processes  evaluated  by  SCE  include  decision-making  processes 
(such  as  project  planning),  communication  processes  (such  as  intergroup 
communication),  and  technical  support  processes  (such  as  peer  reviews 
and  product  engineering) — but  not  technical  production  processes  (i.e. 
processes  required  by  a  particular  methodology,  such  as  object  oriented 
design).  The  SCE  Method  does  nof  evaluate  technical  production  process- 
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es  such  as  requirements  analysis,  specification,  and  design,  but  instead 
focuses  on  the  management  of  the  technical  production  processes  and  on 
other  key  processes,  as  shown  in  Figure  1-1 . 


Organizational  Management  Support 

Examples:  / - 

•  Process  Project  Management  Support 

definition  Examples:  - 

•Process  .Project  (  product B 

focus  planning 

•  Standards  •  Project  Examples. 

•Training  tracking  'Peer  reviews 

•  Software  •  Configuration  * 

quality  management  engineering 

management  -Quality  -Requirements 

management 


assurance 


Product  Building  Operational  Support 

Examples:  . - - - - - 

•Peer  reviews  |  Software  Development  Operations 
•^oduct  Examples:  ■ 

engineering  ,  Engineering  Environments 

equirements  ,  Reqajfements  analysis  methodologies 

management  o  £)gsjg„  methodologies 

•  Code 


Support  for  Support  for  |  ( Support  for  Technical 

decision-making  decision-making  communication  and  processes 

processes  and  communication  technical  processes 

processes 


Evaluated  by  SCE 


Not  Evaluated  by  SCE 


Figure  1-1:  Processes  Evaluated  by  SCE 


SEI  software  process  principles  are  derived  from  the  works  of  Deming,  Ju- 
ran,  and  others  [Deming  86],  [Juran  88],  [Juran  89],  [Crosby  79],  who  pro¬ 
moted  the  idea  that  close  attention  to  the  processes  used  to  create 
products  leads  to  improved  product  quality— i.e.,  the  product  will  fully  sat¬ 
isfy  the  customer’s  requirements  and  will  be  produced  within  existing  con¬ 
straints  such  as  cost  and  schedule.  There  are  many  examples  of  this 
principle  and  its  successful  application  in  the  manufacturing  domain,  but 
the  principle  can  be  applied  anywhere  management  and  communication 
processes  play  an  important  role  in  the  success  of  an  organization’s  mis¬ 
sion. 
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The  SEI’s  "^^Capability  Maturity  Model®*^  (CMM®^)  applies  this  principle 
to  the  software  development  arena.  The  CMM  defines  several  "♦key  pro¬ 
cess  areas  (KPAs);  each  KPA  “identifies  a  cluster  of  related  activities  that, 
when  performed  collectively,  achieve  a  set  of  goals  considered  important 
for  enhancing  process  capability.”^  Each  KPA  contributes  to  the  environ¬ 
ment  in  which  development  organizations  create  software  products.  Within 
the  CMM,  the  KPAs  are  organized  into  five  basic  levels  of  process  maturity 
to  describe  the  progression  from  an  ad  hoc  software  process  to  one  that  is 
well  defined  and  can  act  as  a  stable  foundation  for  continuous  process  im¬ 
provement. 

By  evaluating  the  development  organization’s  software  projects  against 
the  KPAs  in  the  CMM,  the  SCE  team  determines  whether  the  development 
organization  follows  a  stable,  predictable  software  process.  Although  ma¬ 
ture  processes  do  not  guarantee  a  successful  product,  the  likelihood  of 
success  should  increase  as  the  software  processes  mature  toward  the  Op¬ 
timizing  level.  In  other  words,  mature  processes  reduce  the  risk  associated 
with  the  planned  development. 

1 .3  Relationship  to  Other  Documents 

Figure  1-2  shows  an  overall  SCE  product  suite.  SCE  products  can  be 
characterized  by  their  function  served.  They  are  products  necessary  to 

•  transfer  the  SCE  method  to  users  (method  and  guidance), 

•  train  personnel  aboijt  various  aspects  of  SCE,  (education,  training  and 
qualification)  or 

•  install  SCE  as  a  routine  business  practice  of  an  organization  (transition 
and  installation). 

Each  type  of  product  is  useful  by  itself.  However,  all  are  intended  to  form 
a  comprehensive,  integrated  product  suite 


Capability  Maturity  Model  and  CMM  are  service  marks  of  Carnegie  Mellon  University. 

^  Mark  Paulk,  et  al.  Capability  Maturity  Model  for  Software,  Version  1. 1  [Paulk  93a],  page  A- 
10. 


4 


©  1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


CMU/SEI-95-TR-012 


Introduction 


April  1996 


The  SEI’s  '"^Capability  Maturity  Model®^  (CMM®^)  applies  this  principle 
to  the  software  development  arena.  The  CMM  defines  several  "♦key  pro¬ 
cess  areas  (KPAs);  each  KPA  “identifies  a  cluster  of  related  activities  that, 
when  performed  collectively,  achieve  a  set  of  goals  considered  important 
for  enhancing  process  capability.”^  Each  KPA  contributes  to  the  environ¬ 
ment  in  which  development  organizations  create  software  products.  Within 
the  CMM,  the  KPAs  are  organized  into  five  basic  levels  of  process  maturity 
to  describe  the  progression  from  an  ad  hoc  software  process  to  one  that  is 
well  defined  and  can  act  as  a  stable  foundation  for  continuous  process  im¬ 
provement. 

By  evaluating  the  development  organization’s  software  projects  against 
the  KPAs  in  the  CMM,  the  SCE  team  determines  whether  the  development 
organization  follows  a  stable,  predictable  software  process.  Although  ma¬ 
ture  processes  do  not  guarantee  a  successful  product,  the  likelihood  of 
success  should  increase  as  the  software  processes  mature  toward  the  Op¬ 
timizing  level.  In  other  words,  mature  processes  reduce  the  risk  associated 
with  the  planned  development. 

1 .3  Relationship  to  Other  Documents 

Figure  1-2  shows  an  overall  SCE  product  suite.  SCE  products  can  be 
characterized  by  their  function  served.  They  are  products  necessary  to 

•  transfer  the  SCE  method  to  users  (method  and  guidance), 

•  train  personnel  aboijt  various  aspects  of  SCE,  (education,  training  and 
qualification)  or 

•  install  SCE  as  a  routine  business  practice  of  an  organization  (transition 
and  installation). 

Each  type  of  product  is  useful  by  itself.  However,  all  are  intended  to  form 
a  comprehensive,  integrated  product  suite 


SM 

Capability  Maturity  Model  and  CMM  are  service  marks  of  Carnegie  Mellon  University. 

^  Mark  Paulk,  et  al.  Capability  Maturity  Model  for  Software,  Version  1. 1  [Paulk  93a],  page  A- 


4 


©  1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


CMU/SEI-95-TR-012 


April  1996 


Introduction 


Method  and  Guidance 

Method  Description 


SCE  Product  Suite 


I _ 

Education 
and  Quail 

I,  Training, 
fication 

1 

Introduction  to  SCE 


Transition  and 
Installation 

Transition  Strategy 


impiementation  Guide 
for  Suppiier  Selection 

Team  Member  Reference 
Guide 

Quick  Reference  Manual 
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Communication  Package 
Automated  Support  Aids 


Figure  1-2:  SCE  Product  Suite 


Bold  items  in  the  figure  above  reflect  the  specific  pieces  of  the  initial  re¬ 
lease  of  the  SCE  V3.0  product  suite.  Licensed  SEI  providers  of  SCE  ser¬ 
vices  may  provide  additional  items  not  listed  in  bold  above.  With  SEI 
approval,  they  can  become  part  of  the  Product  Suite.  Reference  model 
(e.g.  CMM)  knowledge  and  skill  building  is  essential  to  the  successful  use 
of  SCE.  Related  CMM  products  and  services  that  could  be  offered  to  fulfill 


CMU/SEI-95-TR-012 


©  1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


5 


Introduction 


April  1996 


these  needs  are:  Beginner’s  CMM  Education,  Advanced  CMM  Training, 
CMM  Self-Study  Course,  and  CMM  Test.  The  table  below  depicts  the  pri¬ 
mary  SCE  product  types  and  their  respective  purposes. 


Product  Type 

Purpose 

Method  Description 

Describes  the  Vhaf  aspects  of  the  method.  Provides  a  high 
level  process  description  and  a  baseline  for  all  related 
products,  and  future  improvement. 

Team  Member  Guidance 

Describes  the  “how  to”  aspects  of  the  method.  Provides  a 
detailed  process  description  including  step  by  step 
instructions  for  reliable  usage  and  quick  reference  material  for 
field  use. 

Implementation  Guidance 

Describes  tailoring  aspects  for  different  method  application 
environments.  Provides  cost,  resource,  and  benefit 
information.  Provides  application  specific  templates  for  use. 

Education,  Training,  and  Qualification 

Provides  information  necessary  for  executives,  managers, 
and  practitioners  to  communicate  about  evaluation  use. 
Provides  essential  knowledge  and  skill  building  necessary  to 
successfully  implement  the  method.  Provides  a  basis  for 
evaluator  credibility  and  career  enhancement.  Establishes  a 
necessary  prerequisite  to  certifying  evaluation  results. 

Transition  and  Installation 

Provides  technology  transition  and  change  management 
information  necessary  to  successfully  make  evaluation  use  a 
routine  business  practice  in  an  organization. 

To  provide  both  general  process  concepts,  appraisal  process  descrip¬ 
tion,  and  implementation  details  effectively,  method  material  is  divided 
into  several  components; 

•  a  method  description,  which  defines  basic  “what  and  why”  concepts  for 
the  SCE  method 

•  a  more  detailed  Team  Member's  Reference  Guide  which  describes  the 
“how-to”  aspects  of  the  method.  Additionally,  it  is  a  quick  reference 
manual  with  matrices,  cards,  and  checklists  for  use  by  teams  in  the 
field  during  process  enactment. 

•  implementation  guides  for  each  method  application  type  (such  as  SC  E 
for  supplier  selection,  process  monitoring,  or  internal  evaluation), 
which  describe  application  tailoring  differences.  Users  select  the  guide 
most  appropriate  to  their  environments.  The  initial  SCE  V3.0  release 
includes  the  Implementation  Guide  for  Supplier  Selection. 

•  evaluator  training,  based  on  a  case  study  approach  to  participant 
learning.  Reference  model  (CMM)  knowledge  is  a  prerequisite  to  at¬ 
tending  Evaluator  Training.  Different  case  studies  are  envisioned  for 
use  in  different  environments,  depending  on  the  background  of  the  par¬ 
ticipants.  Formal  training  is  one  component  of  an  overall  evaluator 
qualification  process  that  will  enhance  credibility  of  teams  and  evalua¬ 
tion  results. 
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Part  2  Overview  of  SCE  Implementation 
Guidance 

This  section  provides  an  overview  of  the  guidance  necessary  for  imple¬ 
menting  SCE  V3.0  into  the  typical  acquisition  environment. 

Government  acquisitions  are  a  complex  environment  with  many  different 
processes,  products,  and  players.  Because  every  acquisition  has  unique 
requirements,  it  is  beyond  the  scope  of  this  implementation  guidance 
document  to  provide  specific  information  on  what  is  important  for  every 
variation  of  an  acquisition  (source  selection,  award  fee,  monitoring,  sur¬ 
veillance).  Nevertheless,  the  suggestions  given  in  this  document  provide 
a  solid  basis  for  implementing  SCE  V3.0  into  your  acquisition.  This  doc¬ 
ument  concentrates  on  the  activities  necessary  for  integrating  SCE  V3.0 
into  supplier  selection  application.  Contract  monitoring  activities  are  dis¬ 
cussed  as  appropriate,  but  will  be  the  subject  of  a  separate,  more  exten¬ 
sive  guidance  document. 

2.1  Origin  of  SCE  in  Government  Acquisitions 

Guidance  for  the  use  of  SCE  in  government  acquisitions  has  its  roots  in 
the  Department  of  Defense  Directives  (DoDD)  5000  series.  Specifically, 
the  following: 

DoDD  5000.1: 

Page  1  -6,  item  5  “Competition  and  Source  Selection” 

c.  Contractor’s  past  performance  and  current  capability  (technical... 
managerial)  shall  be  considered  in  source  selection  and  responsibility 
determinations... 

Department  of  Defense  Instruction  (DoDI)  5000.2: 

Page  6-D-1-1  under  “Software  Engineering  Practices” 

b.  Specific  practices  that  should  be  used  are:  (1 )  Establishment  of  a  soft¬ 
ware  process  maturity  model  and  process  improvement  plan... 

Figure  2-1  shows  at  a  high  level  the  context  in  which  the  SCE  method 
interacts  with  and  relates  to  the  major  source  selection  activities  and 
participants  (this  figure  was  adapted  from  the  Software  Development 
Capability  Evaluation  (SDCE)  pamphlet  [AFMC  93]).  Note  that  the  pre¬ 
ponderance  of  the  implementation  guidance  is  based  upon  DoD  usage. 
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It  may  be  useful,  therefore,  to  define  some  of  the  typical  acquisition  ter¬ 
minology  with  an  equivalent  commercial  term.  Table  2-1  below  attempts 
to  set  the  context  of  the  depicted  government  terms  into  terms  more  ap¬ 
plicable  to  the  layperson  or  non-government  person.  See  Appendix  A  on 
page  99  for  formal  definitions  of  the  government  terms. 


Government  Term 

Layperson  Term 

Source  Selection  Authority  (SSA) 

Decision  maker 

"■^Source  Selection  Advisory  Council  (SSAC) 

Committee-  Advisors  charged  with  recommending  action  on 
specific  proposed  alternatives 

"^Source  Selection  Evaluation  Board  (SSEB) 

Action  team  -  chartered  to  evaluate  various  proposed  alternative 
solutions  against  a  predetermined  set  of  standards/evaluation 
criteria 

'"^Evaluation  Teams  (Technical,  Management,  Cost) 

Working  group  -  teams  of  individuals  responsible  for  performing 
analysis  of  differing  segments  of  alternative  approaches  to  a  set 
of  requirements 

'"^Procuring  Contracting  Officer 

Contracts  manager 

Table  2-1:  Government  Acquisition  Terms  vs.  Layperson  Terms 
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Figure  2-1 :  SCE  Method  in  Context 
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2.2  Overview  of  SCE  Applications 

SCE  typically  is  used  in  two  different  environments  within  government 
acquisitions:  source  selection  and  contract  monitoring.  Source  selec¬ 
tion,  the  application  for  which  SCE  was  originally  developed,  has  had  the 
largest  percentage  of  use  since  1987.  Recent  trends,  however,  have 
seen  a  consistent  application  of  SCE  in  the  post-contract  award  environ¬ 
ment;  similarly,  the  commercial  sector  of  the  software  community  has 
been  applying  SCE  in  the  selection  of  subcontractors  and  teaming  part¬ 
ners. 

Factors  to  consider  before  deciding  to  use  SCE  in  an  acquisition  include 
the  following: 

•  criticality  of  an  acquisition  or  the  software  component 

•  total  dollar  value  of  the  acquisition  or  software  component 

•  management  control  priority 

•  unprecedented  system  mission  needs 

•  acquisition  life  cycle  phase 

•  length  of  acquisition  time  period 

•  software  size,  the  number  of  computer  software  components 
(CSCs) 

•  prime  contractor  -  subcontractor  relationship 

These  factors  are  examined  in  detail  in  Section  3.1. 1.1  on  page  25. 

2.2.1  SCE  in  Supplier  Selection 

The  factors  listed  above  affect  the  implementation  of  an  SCE  and  be¬ 
come  visible  in  the  acquisition  documentation: 

•  Commerce  Business  Daily  Announcement 

•  Source  Selection  Plan  (SSP) 

•  Evaluation  Plan  (EP) 

•  Bidder’s  Briefing 

•  "^Request  For  Proposal 

•  possibly,  the  Statement  of  Work  or  Award  Fee  Plan 

•  briefing  to  winning  offeror 

•  briefing  to  losing  offerors 
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When  used  effectively,  virtually  every  major  activity  in  a  source  selection 
is  affected  by  SCE.  Each  of  these  documents,  particularly  the  SSP,  the 
EP  and  the  RFP,  smooths  the  use  of  the  SCE  on  the  contract. 

Figure  2-2  shows  a  global  view  of  a  representative  source  selection 
schedule  that  includes  SCE  activities. 


decision 
to  use  SCE; 

RFP 

proposals 

SSEB  determines 
technical 

briefing  of 

SSP/evaluation 

released 

• 

received 

• 

rating/risk 

• 

SSAC/SSA 

• 

award 

• 

For  each  offeror: 


•  determine/prioritize  focus  areas;  finalize 
data  collection  plan  (1-5  days) 

•  visit  site  (3-5  days) 

•  produce  findings  report  (2^  days) 


Figure  2-2:  SCE  Activities  in  a  Typical  SS  Timeline 


Decision  Point  to 
Proposai  Receipt 


The  decision  to  use  SCE  immediately  sets  things  in  motion  for  appraisal 
planning  and  Implementation.  Nominally,  the  decision  is  articulated  in 
the  Source  Selection  Plan,  and  detailed  usage  of  the  determination  of 
SCE  results  is  delineated  in  the  Source  Selection  Evaluation  Plan.  Ap¬ 
propriate  language  is  selected  and  tailored  for  insertion  in  the  Request 
for  Proposal  specifying  SCE  usage  and  how  the  development  organiza¬ 
tions  are  to  provide  SCE  related  information  for  the  acquisition.  Select¬ 
ing  the  team  leader  and  SCE  team  members  and  training  them  will  occur 
normally  prior  to  proposal  receipt. 


Proposal  Receipt  to  Following  proposal  receipt,  the  evaluation  team  will  determine  the  spe- 

Site  Visit  Qfjp  collection  plan  to  be  carried  out  for  each  development  organi¬ 

zation  remaining  in  the  "^competitive  range  of  the  source  selection. 
The  appraisal  plan  will  have  defined  the  "^organizational  scope  as  well 
as  the  ""t^reference  model  scope  that  are  the  precursors  for  defining  the 
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explicit  data  collection  strategy.  During  the  onsite  period,  the  team  col¬ 
lects  information  and  turns  the  information  into  findings  in  the  form  of 
"^strengths,  "-^weaknesses,  and  —improvement  activities.  A  —rat¬ 
ing,  may  be  determined  if  all  necessary  criteria  are  met  and  the  sponsor 
has  requested  it.  The  data  and  findings  are  then  provided  to  the  spon¬ 
soring  organization  in  the  format  agreed  upon. 

It  is  crucial  that  the  environment  in  which  an  SCE  is  executed  for  acqui¬ 
sition  be  understood.  Normally  in  a  source  selection,  SCE  results  be¬ 
come  part  of  a  Risk  Assessment. 

Note  that  in  most  source  selections  the  SCE  team  is  normally  one  of  a 
number  of  teams  involved  in  providing  evaluation  services  to  the  Source 
Selection  Evaluation  Board  (SSEB).  Typically,  there  will  be  other 
“teams”  evaluating  criteria  in  management,  cost,  and  other  technical  ar¬ 
eas  (Appendix  B  on  page  121  provides  an  example  of  SCE  use  as  a 
technical  criterion).  These  teams  provide  their  findings— just  as  the  SCE 
team  provides  theirs — according  to  the  SSP  and  SSEP. 

For  example,  the  SSEB  evaluates  development  organizations’  propos¬ 
als  for  an  acquisition  relative  to  a  prescribed/published  set  of  evaluation 
criteria  and  identifies  the  risks  (relative  to  the  evaluation  criteria)  of  de¬ 
velopment  organizations  being  able  to  fully  execute  a  contract  if  award¬ 
ed  to  them.  This  risk  assessment  is  provided  to  the  Source  Selection 
Advisory  Council  (SSAC).  The  SSAC’s  responsibility  is  that  of  risk  as¬ 
sessment.  The  SSAC  compares  the  risks  identified  by  the  SSEB  of  each 
development  organization  against  one  another  (e.g.,  organization  A 
compared  to  organization  B,  compared  to  organization  C).  They  assess 
the  overall  risks  of  selecting  one  organization  relative  to  the  other  and 
provide  their  assessment  of  risk  to  the  Source  Selection  Authority  (SSA), 
who  is  empowered  to  make  an  award  of  an  executable  contract.  The 
SSA’s  responsibility  is  to  make  an  award  decision  that  minimizes  risks 
and  maximizes  the  benefits  to  the  sponsoring  governmental  agency. 

2.2.2  SCE  in  Contract  Monitoring 

The  value  of  performing  an  SCE  in  source  selection  can  continue  past 
the  contract  award  and  into  contract  performance.  The  source  selection 
SCE  identifies  a  set  of  risks  associated  with  the  winning  development  or¬ 
ganization.  Those  same  risks,  defined  as  weaknesses  associated  with 
individual  key  process  areas,  can  be  tracked.  Additionally,  these  same 
weaknesses  can  be  monitored  for  improvements  as  the  contract 
progresses.  Monitoring  improvements  can  be  done  by 
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•  using  weaknesses  to  define  the  risks 

•  developing  a  plan  to  mitigate  the  risks 

•  performing  trade  off  analysis  for  surveillance  of  strong  or  weak 
areas 

•  negotiating  Contract  Data  Requirements  List  (CDRL)  trades 
with  the  development  organization  (e.g.,  accept  contractor 
format  for  some  documentation  while  requesting  additional 
status  on  process  improvement  in  selected  KPAs) 

In  contemplating  using  SCE  as  a  contract  process  monitoring  risk  man¬ 
agement  tool  the  following  questions  could  be  considered: 

•  What  would  you  like  (need)  to  know  at  the  start  of  the 
contract? 

•  What  expertise  would  the  program  office  need  to  monitor 
performance? 

•  What  are  the  program  office’s  expectations  about  CDRLs 
(e.g.,  SRS,  SDD)? 

•  What  action  should  be  taken 

•  at  the  start  of  the  contract? 

•  if  identified  risks  occur? 

Use  the  SCE  data  to  define  the  risks  to  execution  of  the  contract,  devel¬ 
op  a  plan  to  mitigate  those  risks  and  work  the  plan.  This  plan  could  entail 
such  items  as  tradilig  off  the  surveillance  of  strong  areas  for  weak  ones. 
If  an  organization  is  found  to  have  excellent  configuration  management 
procedures,  it  is  wasteful  to  check  on  this  process  area  in  the  same  way 
that  would  be  applied  to  an  area  found  to  be  weak  (e.g.,  Software  Project 
Tracking  and  Oversight). 

2.2.3  Using  SCE  to  Baseline  Performance 

SCE  has  been  used  to  “baseline”  contract  process  performance.  One 
strategy  successfully  used  is  baselining  the  development  organization’s 
performance  relative  to  the  CMM  reference  model.  This  entails  a  num¬ 
ber  of  planning  and  execution  factors.  Below  are  two  environments  and 
the  salient  points  to  be  integrated  into  a  plan  for  use  of  SCE. 

For  new  contracts: 

•  RFP  must  identify  SCE  use 

•  SCE  still  an  evaluation  factor  in  selection 
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•  criteria  shouid  be  based  on 

•  eliminating  weaknesses 

•  creating  additional  strengths 

•  improving  actual  versus  planned  tracking  within 
KPAs 

For  existing  contracts  SCE,  as  a  process  monitoring  tool,  can  be  used 

•  as  a  negotiated  contractual  action 

•  when  a  iong-term  relationship  is  expected 

•  with  the  same  criteria  as  for  new  contracts 

Establishing  a  “process  baseiine”  lends  further  utility  of  the  SCE  method 
in  process  monitoring  in  the  situation  of  award  fees  or  for  consideration 
of  a  value  engineering  incentive  for  software  process  improvement. 
Note,  however,  award  fee  applications  (i.e.,  an  award  for  meeting  spec¬ 
ified  measures  of  performance)  are  not  appropriate  in  all  instances.  The 
award  fee  application  of  SCE  is  most  appropriate  when 

•  a  long-term  relationship  is  involved 

•  the  contractor  lacks  a  sufficient  number  of  programs  over 
which  to  spread  improvement  costs 

•  process  investments  in  general  would  not  otherwise  be  made 

•  the  sponsoring  organization  believes  direct  investment 
incentives  wiii  be  the  best  motivator  of  action 

•  the  program  environment  inciudes 

•  mission-criticai  software 

•  software  embedded  throughout  system 

•  history  of  software  issues 

•  long  relationship  with  contractor  expected 

•  SCE  is  used  by  sponsoring  organization  to  mitigate 
risks 

•  the  SCE  appiication  objective  and  ultimate  goal  are  the 
following: 

•  objective:  provide  incentive  for  contractor  to 
improve  the  total  software  development  process 

•  goal:  exceed  the  software  development  quality, 
cost,  and  schedule  requirements 
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The  sponsoring  organization  and  contractor  should  view  themselves  as 
team  members  in  an  effort  to  derive  benefit  from  an  overall  software  pro¬ 
cess  improvement  plan.  This  teaming  approach  has  some  specific  char¬ 
acteristics: 

•  CMM  is  the  basis  for  the  improvement  effort 

•  contractor  uses  CMM  to  establish  plans 

•  sponsoring  organization  evaluates  using  CMM 

•  Contract  incentive  is  the  contractual  vehicle. 

•  describes  sponsoring  organization’s  goals 

•  describes  method  of  evaluating  progress 

•  sponsoring  organization  and  contractor  jointly 
agree  to  criteria  and  approach 

•  Award  fee  plan  increments  and  criteria  are  used  to  help 
achieve  long-range  objectives: 

•  can  tailor  specifically  to  government/contractor 
needs 

•  SCE  is  used  to  baseline  software  process  capability 

•  findings  are  provided  to  the  contractor 

•  contractor  uses  findings  to  focus  the  improvement 
plan 

•  sponsoring  organization  and  contractor  jointly 
agree  to  goals 

•  SCE  is  then  used  to  measure  progress  against  the 
improvement  plan 

•  incentive  awards  are  determined  by  the  contract 
provisions 

•  findings  establish  the  new  baseline  for  the  next 
increment 

The  keys  to  successful  application  of  award  fee  usage  of  SCE  are  to  per¬ 
form  the  source  selection  SCE,  use  the  findings  to  frame  the  award  fee 
plan,  perform  a  baseline  SCE  (after  a  suitable  time  frame  (six  months) 
for  the  contractor  to  begin  contract  performance),  have  the  contractor 
submit  a  software  process  improvement  plan  (SPIP),  and  involve  the 
contractor  to  obtain  the  understanding  of  the  SCE  findings  and  impacts 
upon  the  award  fee  pool. 
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Value  engineering  for  software  process  improvement  is  another  mecha¬ 
nism  available  on  government  contracts.  This  is  described  in  the  Federal 
Acquisition  Regulations  (FAR)  Part  48.  This  FAR  clause  is  extensible  to 
process  improvement.  There  are  five  elements  required: 

1 .  FAR  clause  52.248-1 

2.  Separately  identifiable  software  work  packages  in  an  earned 
value  system 

3.  Baseline  of  prices  for  software  effort 

4.  SCE  to  establish  process  baseline  and  validate  process 
improvements 

5.  Statement  of  Work  (SOW)  requirement  to  develop  process 
improvement  plan  and  support  periodic  SCEs 

What  are  the  advantages?  Exercising  the  value  engineering  clause  has 
a  greater  financial  reward  potential  than  award  fee.  In  addition 

•  award  fee  requires  an  increase  in  government  obligation 
authority;  value  engineering  does  not 

•  however,  value  engineering  requires  visibility  into  software 
work  packages  and  pricing;  award  fee  application  of  SCE 
does  not 

Ultimately,  a  company  exercising  the  value  engineering  clause  has  the 
potential  to  demonstrate  the  software  process  improvement  instantiated 
the  resulting  cost  savings  as  well  as  value  added  to  the  software  prod¬ 
ucts  produced  for  the  sponsoring  organization. 

The  bottom  line  in  the  brief  discussions  of  award  fee  and  value  engineer¬ 
ing  is  that  both  incentivization  approaches  help  management  (sponsor¬ 
ing  organization  and  contractor)  to  focus  on  software  process. 

Table  2-2  provides  the  essentials  of  the  SCE  version  3.0  method  and 
corresponding  source  selection  activities. 

Note  that  much  of  the  material  Table  2-2  duplicates  information  from  the 
SCE  Method  Description.  The  information  is  included  in  this  document 
for  completeness  of  this  overview  section — ^that  is,  to  give  the  reader  a 
brief,  global  view  of  SCE. 
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Each  activity  and  its  relationship  to  acquisition  guidance  from  an  “evalu¬ 
ator  perspective”  and  a  “recipient  perspective”  are  presented  in  Part  3  of 
this  document. 


Activity 

Purpose 

Inputs 

Outputs 

Source  Selection- 
Specific  Activities 

1.  Analyze 
Requirements 

Understand  the 
business  needs, 
objectives,  and 
constraints  to  design 
the  most  appropriate 
appraisal,  and  to 
gain  sponsorship 
and  commitment  for 
the  appraisal. 

Business  context, 

Sponsor  objectives. 

External  schedule, 

Budget, 

Personnel  availability. 
Geographic  constraints, 
Facility  availability. 

Project  availability 

Specific  sponsor  req’ts. 
Program  plans 

Appraisal  goals. 

Appraisal  constraints, 

CMM  scope, 

Organizational  scope. 
Product  profile. 

List  of  appraisal  outputs, 
Sponsor  commitment 

Determine  Requirements 
Initiating  Acquisition 
Planning 

Decision  to  use  SCE 

2.  Develop 
Appraisal 

Plan 

Create  an  artifact  that 
will  guide  and  define 
execution  of  the 
appraisal  such  that  it 
is  In  concert  with 
business  needs  and 
constraints. 

Appraisal  goals. 

Appraisal  constraints, 

CMM  scope, 

Organizational  scope, 

List  of  appraisal  outputs. 
Team  leader  selection 

Initial  appraisal  plan, 
Revised  appraisal  plan 

Sources  Sought 

Commerce  Business 

Daily  (CBD).  Develop 

SSP  Document  how  the 
Source  Selection  will  be 
accomplished.  Write 
Evaluation  Plan  (EP), 

Develop  Request  for 
Proposal  (RFP). 

Note:  The  Following 
implementation  activities 
generally  occur  in 
conjunction  with  SCE 
Method  activities  2  -  4. 
Definitize  SCE  Role  in 
Source  Selection  (e.g., 
specific  criterion,  general 
consideration),  input 
SCE  language  into  RFP 


3.  Select  and 
Prepare  Team 

Have  an 

experienced,  trained 
team  ready  to 
execute  the 
appraisal. 

Appraisal  constraints. 
Product  profile, 

CMM  scope, 
Organizational  scope, 
Appraisal  plan 

Team  leader  selection. 
Team  member  selection. 
Prepared  team 

Development:  SSP,  EP, 

RFP 

SCE  personnel  selected, 
trained  and  prepared  with 
acquisition  requirements 
in  context. 

4.  Obtain 
Organization 
Information 

Obtain  information 
from  the  organization 
that  allows  the  team 
to  refine  the  plan  and 
focus  the 
investigation  in 
subsequent  activities. 

Product  profile, 

CMM  scope, 
Organizational  scope. 
Appraisal  plan 

Request  for  organization 
information, 

Site  information  packet 
(from  organization) 

RFP  issued 

SCE  specific  information 
is  requested  and 
delineated  via  RFP. 

Table  2-2:  SCE  Activities  and  Their  Primary  Inputs  and  Outputs 
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Activity 

Purpose 

inputs 

Outputs 

Source  Selection- 
Specific  Activities 

5.  Analyze 
Instrument 

Data 

Identify  issue  areas 
for  further 
investigation  during 
appraisal  conduct, 
and  to  get  a 
preliminary 
understanding  of  the 
organization’s 
operations. 

Product  profile, 

Site  information  packet 

Questionnaire  response 
summary  analysis, 

Profile  analysis 

Proposal  Receipt 
Evaluation  of  Proposals 
initiated.  Competitive 

Range  Determination. 

Offerors  SCE 
information  analyzed  for 
establishing  “general” 
prioritization  of  reference 
model  components  for 
all  offerors. 

6.  Select  and 
Prepare 
Participants 

Ensure  the  most 
appropriate 
personnel  participate 
in  the  appraisal,  and 
ensure  that  they 
understand  the 
appraisal  process 
and  shared 
expectations. 

Appraisal  plan, 

Site  information  packet. 
Profile  Analyses 

Selected  site(s). 

Selected  project(s). 
Selected  interviewees. 

Initial  briefing  charts, 
Prepared  participants 

Evaluation  of  Proposals 
continues. 

Offerors  projects  are 
selected  to  undergo 

SCE.  Specific 
preparation  for  individual 
offerors  initiated. 

Logistical  coordination 
initiated. 

7.  Prepare  For 
Data 

Collection 

Plan  the  detailed  site 
intervention  to  make 
optimum  use  of 
available  site  visit 
time  to  attain 
appraisal  goals  and 
objectives. 

Appraisal  plan, 

Organization  charts  (from 
site  info  packet), 
Questionnaire  response 
summary  analysis, 

Selected  sites, 

Selected  projects, 

Selected  interviewees 

Data  collection  strategy 

•  interview 

•  document  review 

•  roles/responsibilities 
Interview  questions 

Initial  document  request 
list 

Evaluation  of  Proposals 
continues. 

Prioritization  of 

Reference  Model 
components  and  data 
collection  strategy  for 
onsite  completed. 

Logistical  coordination 
finalized. 

8.  Receive 
Presentations 

Refine/update  the 
appraisal  team's 
understanding  of  the 
organization’s 
software  process 
operations. 

Site  information  packet, 
Appraisal  schedule, 
Development 
organization  presentation 

Updated  site  information, 
Updated  appraisal 
schedule, 

Updated  terminology, 
roles/responsibilities. 
Presentations, 

Working  notes 

Evaluation  of  Proposals 
continues. 

SCE  onsite  for  each 
offeror  is  executed. 

9.  Review 
Documents 

Understand 
processes  actually 
implemented  in  the 
organization. 

Site  information  packet. 
Initial  document  set. 
Document  review 
strategy, 

Annotated 

worksheets/checklists 

Working  notes. 

Requests  for  additional 
documents 

Evaluation  of  Proposals 
continues. 

SCE  onsite  for  each 
offeror  is  executed. 

10.  Conduct 
Interviews 

Understand  site 
personnel 
perspective  on 
processes 

Implemented  in  the 
organization. 

Appraisal  schedule. 
Interview  strategy, 

Site  information  packet. 
Interview  questions, 
Working  notes. 

Annotated 

worksheets/checklists. 
Requests  for 
additional/new 
documents 

Working  notes. 

Requests  for  additional 
documents, 

Requests  for 

additional/new 

interviewees 

Evaluation  of  Proposals 
continues. 

SCE  onsite  for  each 
offeror  is  executed. 

Table  2-2:  SCE  Activities  and  Their  Primary  Inputs  and  Outputs  (Continued) 
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Activity 

Purpose 

Inputs 

Outputs 

Source  Selection- 
Specific  Activities 

11.  Consolidate 

Transform  the  data 

Working  notes, 

Observations 

Evaluation  of  Proposals 

Data 

collected  into  formal 
team  observations  of 
process  strength  and 
weaknesses  relative 
to  the  reference 
model  (e.g.,  the 

CMM). 

Annotated 

worksheets/checklists 

•  CMM 

•  Non-CMM 

Revised  data  collection 
plan 

•  document  review 
strategy 

•  Interview  strategy 
Annotated 

worksheets/checklists 
Requests  for 
additional/new 
interviewees  or 
documents 

continues. 

SCE  onsite  for  each 
offeror  is  executed. 

12.  Deliver  Draft 

Validate  preliminary 

Annotated 

Working  notes, 

Evaluation  of  Proposals 

Findings 

team  observations, 
build  credibility  in  the 
appraisal,  and 
generate  buy-in  to  the 
eventual  results. 

worksheets/checklists 
•  Observations 

Draft  findings 
presentation 

continues. 

SCE  onsite  for  each 
offeror  is  executed.  A 
decision  to  conduct  this 
activity  is  made  during 
Activity  2  Develop 

Appraisal  Plan.  Typically, 
this  activity  will  NOT  be 
conducted  during  a 

Source  Selection  SCE. 

13.  Make  Rating 

Make  decisions, 

Annotated 

Ratings  of  CMM 

Evaluation  of  Proposals 

Judgments 

based  on  validated 
observations,  about 
the  organization’s 
process  capability, 
using  the  reference 
model  components. 

worksheets/checklists, 
Working  notes 

components 

continues. 

SCE  onsite  for  each 
offeror  is  executed. 

14.  Deliver  Final 

Provide  a  clear  and 

Annotated 

Final  findings 

Evaluation  of  Proposals 

Findings 

actionable 
summation  of  the 
appraisal  results  to 
the  organization. 

worksheets/checklists 

presentation, 

•  global  findings 

•  final  findings 

•  non-CMM  findings 

•  ratings 

continues. 

SCE  onsite  for  each 
offeror  is  executed. 

Table  2-2:  SCE  Activities  and  Their  Primary  Inputs  and  Outputs  (Continued) 
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Activity 

Purpose 

Inputs 

Outputs 

Source  Selection- 
Specific  Activities 

15.  Produce 

Produce  a  fonnal 

Appraisal  plan, 

Appraisal  reports 

Source  Selection 

Reports  and 

baseline  of  the 

Site  information  packet, 

•  findings 

Evaluation  Board  (SSEB) 

Support 

appraisal  conduct 

All  presentations 

•  outcomes 

compares  data  collected 

Follow-On 

and  results  for  the 

•  Initial  briefings, 

•  appraisal  data 

against  Evaluation 

Activities 

sponsor  and  other 

•  Organization, 

•  method  evaluation 

Standard-  assigns 

stakeholders,  and 

•  Draft  findings. 

Disposition  of  data 

technical  rating  and  risk 

ensure  the  appraisal 

•  Final  findings, 

Determination  of  post 

identification 

results  are  used 

All  annotated 

appraisal  activities 

Source  Selection 

appropriately  to 

worksheets/checklists. 

Advisory  Council 

achieve  stated 

All  working  notes 

compares  and  ranks 

business  objectives. 

offeror  proposals  submits 

Risk  Assessment  to  SSA 
Source  Selection 
Authority  makes  award 
decision. 

SCE  findings/outcomes 
are  submitted  to  SSEB. 
SCE  Team  consults  with 
SSEB  if  requested.  SCE 
team  may  act  as  advisors 
to  SSAC  and  SSA. 


Table  2-2:  SCE  Activities  and  Their  Primary  Inputs  and  Outputs  (Continued) 
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This  section  examines  the  SCE  activities  individually  from  an  acquisition 
context  perspective.  The  focus  is  on  integrating  the  method  activities 
with  the  acquisition  application  of  the  method.  SCE  application  in  the  ac¬ 
quisition  environment  typically  takes  two  forms,  source  selection  and 
post  contract  award  “process  monitoring.”  The  principal  thrust  of  the  dis¬ 
cussions  that  follow  is  on  the  source  selection  application.  The  “process 
monitoring”  application  is  the  subject  of  a  separate  document,  but  where 
there  is  a  direct  tie  into  the  supplier  selection  activity  as  well  as  the  SCE 
implementation  and  the  “process  monitoring”  application,  appropriate 
comments/guidance  are  provided. 

The  discussion  will  reflect  the  following  structure: 

Tables  illustrating  the  generic  SCE  activity’s  purpose,  actions,  and  out¬ 
comes  compared  with  the  Supplier  Selection  SCE’s  purpose,  actions 
and  outcomes  begin  each  Activity  section.  Discussion  of  implementing 
activities  from  an  evaluator  and  recipient  perspective  follow  these  tables. 
Note  that  in  many  cases  the  Supplier  Selection  SCE  section  of  the  table 
will  contain  “Same  as  Generic  SCE.”  This  demonstrates  the  overall  ap¬ 
plicability  of  the  SCE  method.  The  bulk  of  the  implementation  differenc¬ 
es  of  using  SCE  by  an  organization  with  or  without  formal  acquisition 
rules  occur  in  appraisal  planning  and  use  of  the  SCE  results. 
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3.1  Activity  1  Analyze  Requirements 

Table  3-1  provides  an  overview  of  this  activity. 


1  Generic  SCE 

1  Supplier  Selection  SCE  I 

Purpose 

Understand  the  business  needs,  objectives, 
and  constraints  to  design  the  most 
appropriate  appraisal,  and  to  gain 
sponsorship  and  commitment  for  the 
appraisal. 

Understand  the  sponsor's  business  needs, 
objectives,  and  constraints  in  incorporating 
the  SCE  methodology  into  the  acquisition  and 
the  sponsorship  necessary  for  Integration  and 
execution  of  the  methodology. 

Actions 

Develop  appraisal  goals,  constraints,  and 
scope. 

Determine  what  appraisal  outputs  will  be  and 
how  they  will  be  used. 

Same  as  generic  SCE 

Initiate  acquisition  planning 

Decide  whether  to  use  SCE 

Outcome 

The  decision  to  proceed  with  the  appraisal, 
based  on  a  shared  understanding  of  the 
evaluation  goals,  constraints,  and  scope. 

Commitment  to  integrate  SCE 
into  the  acquisition. 

Table  3-1 :  Analyze  Requirements  Overview 


3.1.1  Evaluator 

The  central  question  to  be  answered  is  whether  software  capability  in¬ 
formation  is  needed  or  desired  to  proceed  with  the  acquisition. 

Determining  How  to  Most  organizations  selecting  suppliers  of  software  products  and  servic- 

Use  SCE  es  would  prefer  to  have  more  discriminators  available  other  than  cost 

and/or  the  technical  performance  promised  in  a  proposal.  Use  of  SCE 
can  provide  such  a  discriminator  among  the  various  software  develop¬ 
ment  organizations  that  are  under  consideration.  Table  3-2  and  Figure 
3-1  demonstrate  the  need  for  some  analysis  of  the  requirements  at  the 
outset.  Note  that  the  decision  diamond  (Figure  3-1  on  page  24)  labelled 
Source  Selection,  Teaming,  or  Subcontracting,  considers  three  acquisi¬ 
tion  applications.  The  sections  that  follow  will  discuss  SCE  with  respect 
to  a  “yes”  answer  to  this  decision  diamond  in  relation  to  the  perspective 
of  deciding  to  use  or  not  use  the  SCE  Method.  The  “no”  option  leading 
to  a  “yes”  regarding  incentivization  will  be  discussed  as  a  separate  ap¬ 
pendix  to  this  document. 

Source  Selection:  This  is  the  traditional  use  of  an  SCE.  This  application 
typically  has  referred  to  the  application  whereby  a  U.S.  government 
agency  (e.g..  Army,  Navy,  Air  Force,  Coast  Guard,  Federal  Aviation  Ad¬ 
ministration)  uses  SCE  as  a  contract  award  evaluation  criterion.  For  a 
company  contemplating  the  selection  of  suppliers  of  software  services 
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or  products  (non-government  procurement)  many  of  the  goals,  objec¬ 
tives,  constraints,  strategies,  and  planning  factors  are  germane  and 
should  be  considered. 

Teaming:  The  teaming  situation  refers  to  the  industry  practice  of  select¬ 
ing  partners  or  “teammates”  to  collaboratively  compete  for  contracts, 
government  or  commercial.  This  a  variation  of  the  source  selection  ap¬ 
plication  use  of  SCE.  For  this  application,  the  degree  of  legal  formality 
may  typically  be  reduced.  However,  the  considerations  for  selecting 
such  a  “teammate”  remain  essentially  the  same. 

Subcontracting:  The  subcontracting  instance  refers  to  the  situation 
whereby  one  entity  (e.g.,  government  prime  contractor,  or  commercial 
entity)  establishes  a  relationship  with  another  entity  for  the  purposes  of 
receiving  services  or  products  that  will  then  be  applied  to  an  existing  ef¬ 
fort.  This  could  take  the  form  of  the  primary  organization  “subcontract¬ 
ing”  a  specified  portion  of  an  existing  contract,  statement  of  work  or  for 
separately  Identified  services  that  will  supplement  its  own  efforts  on  a 
contract  or  product. 
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3.1 .1 .1  Criteria  for  Using  SCE  in  Supplier  Selection 

General  familiarity  with  the  acquisition  organization  source  selection 
process  is  assumed  for  purposes  of  focusing  on  each  participant’s  (eval¬ 
uator  or  recipient)  relationship  to  SCE.  Clearly,  SCE  should  not  be  used 
for  every  software  acquisition.  Both  the  costs  and  benefits  of  using  SCE 
as  well  as  the  specific  nature  of  the  acquisition  should  be  considered 
when  making  this  decision.  These  costs  and  benefits  may  indicate  that 
other  approaches  are  necessary  for  very  small  acquisitions  involving 
software.  See  Appendix  C  for  current  government  policies  regarding  the 
use  of  SCE  in  acquisitions.  This  section  discusses  criteria  related  to  the 
nature  of  an  acquisition. 

There  is  no  minimum  number  of  lines  of  code  or  measure  of  software  in¬ 
tensity  dictating  that  SCE  must  be  used  in  an  acquisition.  Several  con¬ 
siderations  must  be  weighed  by  a  program  manager  when  making  the 
decision.  Each  of  the  following  factors  are  important  considerations,  but 
the  program  manager  or  person  responsible  for  determining  SCE  usage 
for  an  acquisition  must  weigh  these  factors  in  accordance  with  the  orga¬ 
nization’s  method  of  procuring  systems  and  services.  These  are  general 
guidelines  that  must  be  refined  to  meet  the  context  of  the  organization: 

•  criticality  of  an  acquisition  or  the  software  component 

•  total  dollar  value  of  the  acquisition  or  software  component 

•  management  control  priority 

•  unprecedented  system  mission  needs 

•  acquisition  life  cycle  phase 

•  length  of  acquisition  time  period 

•  software  size,  the  number  of  CSCs 

•  prime  contractor  -  subcontractor  relationship 

Table  3-2  illustrates  the  relationship  of  each  of  these  factors  as  a  general 
guideline  for  determining  appropriateness  of  SCE  usage.  Each  box 
should  be  read  independently,  and  then  combined  with  other  factors,  to 
make  an  overall  judgement  on  SCE  applicability. 
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Critical 

Software 

Dollar 

Value 

Manage¬ 

ment 

Control 

Software 

Prece¬ 

dence 

Lifecycle 

Phase 

Schedule 

Length 


Software 

Size 


1  Decision  I 

Definitely  Use  SCE 

Strongly  Consider 
Using  SCE 

Consider  Using 

SCE 

SCE  Use  Likely  1 

Not  Appropriate  1 

Major  government 
program  MCCR 
systems 

Non-MCCR  systems 

Software  >  $5M 

Total  program  >  $25M 

Total  program  >  $10M 

•  Software  <  $5M; 

<  30%  of  total  cost 

•  Total  EMD  <$10M 

High  Priority 

Low 

priority 

Unprecedented 

system 

Need  defined,  any 
software  CSCIs 
unprecedented 

Precedented  system 

EMD  phase 

•  Dem/Val 

•  concept 
exploration 

Operational 
readiness  support 

Production/ 

deployment 

Upgrades,  major 
modifications  or 
follow-ons  expected 

•  Development  >  24 
months 

•  System  life  >  10 
years 

Program  length 
>  5  years 

>  100  KSLOC 

>  50  KSLOC 

>  25  KSLOC 

<  25  KSLOC 

Table  3-2:  SCE  Usage  Decision  Making  Criteria 


Criticality  of  an  Acquisition  or  the  Software  Component 

The  criticality  of  an  acquisition  may  necessitate  SCE  use.  The  SEI  rec¬ 
ommends  that  any  government-defined  “major”  program  use  SCE  as  an 
integral  part  of  its  strategy  for  producing  the  highest  quality  end  product 
and  motivating  government  contractors  to  focus  on  software  process  im¬ 
provement  as  a  means  to  effect  this  goal.  In  all  Mission  Critical  Comput¬ 
er  Resource  (MCCR)  systems,  regardless  of  total  dollar  amount, 
software  size,  or  DoD  priority  ranking,  SCE  use  should  be  strongly  con¬ 
sidered.  MCCR,  and  software  in  general,  are  critical  components  of 
modern  weapon  systems.  The  success  of  the  system  is  largely  depen¬ 
dent  upon  software  precisely  performing  its  intended  function.  An  exam¬ 
ple  of  a  small,  but  highly  critical  piece  of  software  warranting  the  use  of 
SCE  in  an  acquisition  would  be  software  needed  to  control  the  hardware 
for  an  access  control  system  for  nuclear  weapons  or  other  munitions. 
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Total  Dollar  Value  of  the  Acquisition  or  Software  Component 

Dollar  value  is  important  because  of  the  investment  in  resources  and 
time  necessary  to  implement  SCE  effectively.  Use  of  SCE  should  be 
considered  when  the  total  value  of  an  acquisition  software  component 
exceeds  $10  million.  On  any  acquisition  in  which  total  cost  is  greater 
than  $25  million,  SCE  use  should  be  strongly  considered. 

Where  the  software  component  of  a  program  itself  exceeds  $5  million  or 
is  greater  than  30%  of  the  total  program  cost,  SCE  should  be  used.  This 
criterion  often  may  take  precedence  over  the  $25  million  threshold  de¬ 
scribed  above.  On  the  other  hand,  some  acquisitions  in  the  $10  to  $25 
million  range  may  not  warrant  the  use  of  SCE  because  of  program- 
unique  circumstances.  Perhaps  the  software  component  is  not  mission 
critical  and  is  less  than  10%  of  the  total  dollar  value.  These  guidelines 
are  not  absolute:  they  are  intended  to  aid  the  decision-making  process 
and  the  development  of  appropriate  policies  and  procedures. 

Management  Control  Priority 

When  management  control  is  a  high-priority  concern,  SCE  use  should 
be  considered.  An  environment  under  effective  management  control  will 
be  more  able  to  produce  data  that  is  useful  for  lessons  learned  which 
can  be  incorporated  into  the  overall  system  development  process. 
These  lessons  help  the  acquisition  organization  avoid  “re-inventing  the 
wheel.”  Successful  management  control  also  facilitates  effective  imple¬ 
mentation  of  modern  methodologies,  tools,  and  techniques. 

A  controlled  environment  is  essential  to  managing  contractor  process¬ 
es — processes  for  maintaining  the  development  environment,  bringing 
new  people  and  technology  into  the  environment,  identifying  problems 
early  in  the  contract,  and  managing  requirements  changes.  A  controlled 
environment  enables  improved  risk  assessment  and  abatement. 

System/Software  Precedence 

SCE  should  be  used  when  the  contractor  is  likely  to  develop  software  im¬ 
plementations  that  are  unprecedented.  Unprecedented  systems  (i.e., 
those  solving  new  or  unique  problems)  pose  special  problems  for  soft¬ 
ware  development  organizations.  An  SCE  of  each  offeror  would  identify 
whether  the  requisite  controls  are  in  place  on  the  contractors’  existing 
programs  and  whether  they  will  be  easily  transferred  to  the  new,  unprec¬ 
edented  system. 
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When  the  mission  of  the  system  system/software  component,  especially 
the  role  played  by  the  software  component,  is  known  and  defined  by  the 
end  user,  and  portions  of  the  system  will  be  unprecedented,  use  of  SCE 
should  be  strongly  considered.  SCE  helps  identify  program  risks  associ¬ 
ated  with  the  capability  of  contractors  to  succeed  in  producing  quality 
software  in  an  unprecedented  environment. 

Use  of  SCE  yields  information  about  an  organization’s  ability  to  manage 
risks  inherent  in  unprecedented  software  development,  as  well  as  its 
ability  to  manage  tasks  which  are  new  but  are  similar  to  ones  they  have 
successfully  completed  previously. 

Acquisition  Lifecycle  Phase 

The  lifecycle  phase  of  an  acquisition  is  an  important  factor  in  determin¬ 
ing  SCE  usage.  The  SCE  Method  and  CMM  were  originally  developed 
in  response  to  DoD’s  and  industry’s  recognized  problems  in  managing 
the  development  of  increasingly  complex  mission  critical,  software-in¬ 
tensive  products  in  the  real-time,  embedded  domain.  Given  this  back¬ 
ground,  SCE  fits  in  any  engineering  manufacturing  development  (EMD) 
program  within  this  domain,  since  EMD  is  the  typical  phase  associated 
with  major  new  software  development.  The  SEI  recommends  that  any 
EMD  program  consider  SCE  use,  in  accordance  with  the  other  factors 
listed  here.  However,  SCE  use  is  not  limited  to  the  EMD  phase.  The  SCE 
Method  has  been  used  successfully  in  demonstration/validation,  con¬ 
cept  exploration,  and  jbperational  readiness  support  phases. 

Length  of  Acquisition  Time  Period 

The  SCE  Method  should  be  considered  in  any  procurement  where  soft¬ 
ware  is  a  major  component  and  the  program  duration  period  is  expected 
to  be  greater  than  24  months.  This  time  frame  is  recommended  because 
of  the  amount  of  resources  necessary  to  apply  SCE  effectively,  and  be¬ 
cause  the  typical  process  improvement  program  implemented  by  a  con¬ 
tractor  requires  at  least  18-24  months  to  attain  and  sustain 
improvements  in  process  maturity.  Thus,  more  software  development 
time  is  necessary  to  see  improved  results  directly  on  the  contract. 

SCE  should  also  be  used  when  the  program  office  expects  significant 
block  upgrades,  modifications,  or  follow-on  programs  to  occur,  and  the 
original  contractor  is  expected  to  be  a  primary  offeror  or  likely  performer 
of  the  new  work.  Often,  the  processes  put  in  place  by  the  contractor  at 
the  start  of  a  software  development  will  be  frozen,  meaning  that  process 
changes  will  be  limited  during  that  development  period.  Software  up- 
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grades  or  major  modifications  to  existing  systems  are  good  times  to  un¬ 
freeze  the  current  software  deveiopment  process  and  install  new, 
improved  processes,  methods,  and  technology.  Therefore,  using  SCE 
during  the  initial  software  development  and  the  subsequent  improve¬ 
ment  programs  will  enable  any  improved  processes  to  be  implemented 
on  the  follow-on  developments. 

SCE  use  may  still  be  appropriate  even  if  neither  of  these  criteria  is  met 
and  the  government  Program  Executive  Officer  (PEO),  center/com¬ 
mander  or  activity  committee  is  attempting  to  motivate  and  gain  im¬ 
provements  in  a  particular  domain  area,  such  as  avionics  systems. 
These  PEO  decisions  may  entail  long-range  considerations  that  go  be¬ 
yond  the  current  contract,  and  thus  SCE  use  may  be  appropriate  to  meet 
other  government  objectives. 

Software  Size,  Number  of  Computer  Software  Components  (CSCs),  and  the  Prime  Contrac¬ 
tor-Subcontractor  Reiationship 

The  amount  of  software  to  be  developed  will  directly  affect  the  number 
of  CSCs  required  to  effectively  partition  the  software  system  into  man¬ 
ageable  chunks  and  the  likelihood  of  a  prime  contractor  performing  inte¬ 
gration  of  software  produced  by  several  subcontractors.  When  the 
estimated  size  of  the  software  component  exceeds  1 00  thousand  source 
lines  of  code  (KSLOC),  SCE  should  be  used.  At  this  threshold,  the  com¬ 
plexity  of  the  software  development  will  be  a  significant  concern  of  the 
program  manager.  Scaling  up  small  software  engineering  teams  to  meet 
the  challenges  of  a  large  development  creates  additional  pressures  on 
effective  software  development  processes. 

When  below  this  threshold,  there  are  several  related  considerations  that 
should  also  be  weighed  by  the  program  manager  when  determining 
whether  to  use  SCE: 

•  Software  size  between  25  and  1 00  KSLOC 

•  Minimum  development  team  of  20  to  100  people  in  under  a 
year  with  several  years  of  support  and  enhancements 

•  Software  embedded  on  multiple  platforms  in  different 
languages  for  a  real-time  application 

•  Highly  specialized  technoiogies:  for  example,  radar  signal 
processing  on  a  unique  programmable  signal  processor  or 
image  processing  on  a  customized  paraliei  processor 

•  Software  pieces  to  be  subcontracted  to  geographicaliy  distant 
locations 
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These  examples  highlight  different  managerial/technical  capabilities  a 
contractor  must  possess  depending  on  the  type,  complexity,  and  size  of 
the  software  and  the  nature  of  the  delivery  schedule. 

A  clear  understanding  of  acquisition-specific  circumstances,  rather  than 
knowledge  of  hard  criteria,  is  necessary  to  determine  whether  SCE  use 
is  appropriate.  In  general,  for  all  acquisitions  with  a  software  component, 
the  acquisition  organization  should  seek  to  do  business  with  contractors 
who  understand  and  effectively  implement  modern  software  develop¬ 
ment  practices,  and  also  with  those  contractors  taking  actions  to  im¬ 
prove  these  practices.  SCE  is  a  tool  that  can  augment  other  acquisition 
organization  techniques  for  discerning  differences  in  the  capabilities  of 
offerors. 

Several  organizations  have  established  criteria  for  SCE  use  which  re¬ 
flect  the  individual  needs  of  these  organizations,  and  supplement  the  in¬ 
formation  contained  in  this  guide.  One  major  military  command  drafted 
a  policy  requiring  SCE  use  on  all  MCCR  programs  exceeding  $10  mil¬ 
lion.  Another  military  division  is  taking  the  approach  of  requiring  SCE 
use  on  procurements  which  include  software  size  estimates  greater  than 
50  KSLOC.  One  commercial  company  is  requiring  its  subcontractor  to 
undergo  a  SCE  and  develop  an  action  plan  to  correct  weaknesses. 
These  examples  underscore  the  importance  of  refining  SCE  usage  cri¬ 
teria  to  best  reflect  the  acquisition  practices  implemented  at  a  particular 
acquisition  organization,  both  government  and  industry.  Different  orga¬ 
nizations  have  different  business  bases,  contractor  communities,  prod¬ 
uct  types,  application  domains,  etc.,  all  of  which  affect  site-specific 
implementing  instructions  for  SCE. 

3.1 .1 .2  Benefits  of  Using  SCE  in  Supplier  Selection 

Use  of  the  SCE  Method  in  Army,  Navy,  Air  Force  and  non-DoD  agencies 
indicates  that  SCE  helps  the  acquiring  organization  in  multiple  ways: 

•  Added  software  development  capability  realism  in  the  source 
selection  process 

•  Increased  objectivity  in  information  collected  for  an  acquisition 

•  Motivation  for  contractor  software  process  improvement 
actions 

Software  Development  Capability  Realism:  One  benefit  SCE  pro¬ 
vides  is  the  software  development  capability  realism  introduced  into  the 
proposal  review  and  contractor  analysis  process.  The  information  SCE 
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collects  is  timely,  real,  and  is  based  on  current  projects  and  the  practices 
actuaiiy  being  implemented  by  offerors’  engineering  and  manageriai 
personnel. 

For  moderate  to  large  software  development  efforts,  a  currently  popular 
means  of  evaluating  a  contractor’s  software  development  abilities  during 
a  source  selection  is  the  review  of  the  offeror’s  Software  Development 
Plan  (SDP).  Comparing  the  SCE  findings  with  the  evaluation  of  the  con¬ 
tractor’s  proposal  and  SDP  will  clarify  for  the  program  office  whether  the 
proposed  approach  is  realistic  in  light  of  the  offeror’s  current  process  ca¬ 
pability.  Based  on  this  comparison,  the  program  office  can  better  evalu¬ 
ate  the  risks  posed  by  each  offeror  and  work  with  the  winning  contractor 
on  a  realistic  software  process  improvement  program. 

Objectivity:  A  second  major  benefit  of  SCE  is  the  objectivity  it  introduc¬ 
es  into  the  proposal  review  process.  The  SCE  Method  helps  ensure  an 
objective  review  by  putting  a  trained  evaluation  team  on  site  to  evaluate 
the  offeror’s  activities  and  compare  them  against  a  public  "♦reference 
model  or  standard  (e.g.,  the  CMM).  In  the  typical  source  selection,  eval¬ 
uating  software  risk  is  a  difficult  task  because  there  are  few  avenues  for 
addressing  this  issue  other  than  by  evaluating  what  is  in  the  proposal. 

With  the  goals  of  the  CMM  KPAs  as  a  basis,  contractor  software  process 
capability  can  be  reliably  measured  against  a  common  standard.  This 
permits  consistent,  repeatable,  and  fair  evaluation  of  contractor  software 
process  capability.  This  adds  value  to  the  source  selection  process  by 
making  software  reviews  more  objective. 

3.1. 1.3  Cost  of  Using  SCE 

Using  SCE  requires  personnel  and  financial  resources,  on  both  the  con¬ 
tractor  and  acquisition  agency  sides.  The  resource  considerations  af¬ 
fecting  the  implementation  of  SCE  are: 

•  Personnel 

•  Time 

•  '"♦Rating  Baseline  (scope  of  evaluation:  one  of  three  options 
selected) 

•  Financial 

•  Development  organization’s  resource  requirement 
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Determining  the  Rating  Baselining 

An  important  aspect  of  the  appraisal  requirements  and  planning  pro¬ 
cess — determining  the  rating  baseline  option  needs  to  be  understood  by 
all,  as  the  decisions  made  impact  all  aspects  of  the  SCE.  Figure  3-2  rep¬ 
resents  the  three  options  available  to  an  organization  contemplating  the 
use  of  SCE. 

The  option  selected  is  directly  related  to  determining  reference  model 
scope,  and  will  drive  the  rest  of  the  appraisal  process,  including  Activity 
13,  Make  Rating  Judgments.  The  rating  baseline  decision  affects  the 
time,  labor,  and  cost  constraints  determined  in  Activity  2,  Develop  Ap¬ 
praisal  Plan.  A  decision  to  rate  "♦maturity  level  based  on  full  coverage 
will  require  additional  resources  relative  to  the  more  “standard”  SCE  ap¬ 
proach  of  selecting  a  subset  of  model  components. 


Depth-Oriented 


Full  model  scope  - 
two  sub-options 

•  full  coverage  of  items,  or 

•  coverage  factor  rating 
without  full  coverage 
Maturify  level  rating 

is  feasible  in  both  sub-options 


Selected  model  components 

•  based  on  subset  of  model 
components 

•  items  rated  only  when  fully  covered 

•  decision  to  rate  items  made  in 
planning 

•  maturify  level  rating  is  not  feasible 


Figure  3-2:  SCE  Rating  Baseline  Options 


32 


©1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


CMU/SEI-95-TR-012 


April  1996 


Activity  1  Analyze  Requirements 


Figure  3-3  on  page  34  illustrates  the  selection  of  the  Depth-Oriented  op¬ 
tion  from  Figure  3-2  and  shows  the  estimated  acquisition  agency  labor, 
in  person  days,  required  to 

•  implement  SCE  in  program  documentation 

•  train  SCE  team  members 

•  conduct  the  site  visits 

•  incorporate  the  SCE  findings  into  source  selection  results/ 
decisions 

The  estimate  assumes  a  single  source  selection,  a  program  office  hav¬ 
ing  no  prior  experience  with  SCE,  and  three  offerors  within  the  compet¬ 
itive  range  who  must  be  evaluated.  For  a  team  of  five  who  conduct  three 
site  visits,  the  total  labor  is  approximately  115  person  days.  For  refer¬ 
ence,  the  estimated  labor  for  an  acquisition  involving  only  one  site  visit 
is  65  person  days  (Total  Effort  Fixed  Costs  39.75  person-days  plus  Vari¬ 
able  Cost  Effort  of  25  person-days  for  site  visit).  Certainly,  there  are 
economies  of  scale  and  there  are  many  non-recurring  costs,  such  as 
team  training,  which  will  continue  to  reduce  overall  acquisition  agency 
labor  costs  as  trained  resources  are  utilized  on  subsequent  acquisitions. 
In  an  instance  where  the  Program  Manager  (PM)  and  SCE  team  have 
been  trained  and  have  used  the  method  previously,  and  the  ""^Procur- 
ing  Contracting  Officer  (PCO)  is  SCE  knowledgeable,  the  estimated 
labor  to  implement  SCE  on  an  acquisition  (with  3  site  visits)  declines  to 
83.5  person  days  (114.25,  less  SCE  information  gathering  5.25,  less 
RFP  preparation  1,  less  SCE  Training  25). 

This  analysis  leaves  it  up  to  members  of  the  individual  program  office  to 
determine  their  own  average  person  cost  per  day,  average  travel  and 
per  diem  costs,  and  subsequently  add  these  to  the  cost  of  team  training 
to  estimate  a  total  dollar  cost  for  implementing  SCE. 
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SCE  Effort 
Phase 

Who  Does  It 

Effort  days  per 
person 

Number  of 
People 

Total  Effort  1 

SCE 

PM,  PCO,  SCE 

0.25 

3 

0.75 

Information 

team  lead 

1.5 

3 

4.5 

gathering 

RFP 

SCE  team 

1.0 

1 

1.0 

Preparation 

leader  or  PM 

1.0 

1 

1.0 

0.5 

1 

0.5 

Fixed 

0.5 

1 

0.5 

Costs  SCE  Training 

SCE  team 

5.0 

5 

25.0 

members 

SCE  Findings 

SCE  team 

1.0 

5 

5.0 

Preparation 

members 

Contractor 

PM.  PCO,  SCE 

0.5 

3 

1.5 

Debriefs 

team  leader 

Subtotals 

11.25 

39.75 

Variable  SCE  Site 

SCE  team 

5.0 

5 

75.0 

Cost  Visits  3 

members 

Total  Person  Days  Effort 

114.75 

Figure  3-3:  Estimated  SCE  Labor  For  One  Source  Selection 


Constraints 

Personnel  Constraints:  The  largest  constraint  on  the  acquisition  agen¬ 
cy  is  the  labor  effort  expended  by  the  individuals  constituting  the  SCE 
team.  This  team  is  needed  for  one  full  work  week  for  every  SCE  site  visit 
that  is  performed.  In  addition,  several  person-days  are  needed  to  pre¬ 
pare  for  each  site  visit  and  prepare  the  detailed  report  or  set  of  findings 
that  is  submitted  to  the  management  body  sponsoring  the  evaluation. 

In  addition  to  the  site  visit  requirements,  the  SCE  team  leader  or  the  pro¬ 
gram  office’s  software  focal  point  will  be  needed  on  a  part-time  basis  pri¬ 
or  to  the  site  visits  to  incorporate  appropriate  language  into  the  source 
selection  materials  that  are  affected  by  SCE,  assemble  an  SCE  team, 
and  schedule  training  for  any  untrained  team  members.  This  part-time 
task  will  be  minimal  once  the  acquisition  organization  has  put  in  place 
support  materials  for  SCE,  including  this  guide.  After  the  site  visits,  the 
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SCE  team  leader  will  likely  be  needed  to  advise  the  evaluation  sponsor 
and  perform  outbriefs  to  the  development  organizations  as  directed  by 
his  or  her  PCO. 

Time  Constraints;  The  SCE  team  is  needed  for  at  least  one  and  a  half 
weeks  for  every  a  site  visit.  This  includes 

•  Preparation:  1-2  days 

•  Travel  time:  1  day 

•  Site  visit:  3  -  Sdays  (includes  caucus  and  findings  preparation) 

•  Time  off  between  site  visits:  1  day 

Time  off  is  important  because  site  visits  are  intense  activities.  Another 
time  constraint  is  imposed  by  the  typical  source  selection  schedule.  Site 
visits  cannot  begin  until  after  the  initial  proposal  evaluation  and  only  on 
those  offerors  remaining  in  the  competitive  range.  This  typically  allows  a 
one  to  two  month  window  to  conduct  the  on-site  phase  of  the  SCE.  A 
program  manager  does  not  know  the  number  of  offerors  until  proposals 
are  received.  This  means  that  the  program  manager  will  have  to  esti¬ 
mate  how  much  time  is  needed  to  complete  all  the  SCEs  based  on  the 
estimated  number  of  offerors. 

Financial  Constraints:  Given  a  $10  million  acquisition,  which  was  intro¬ 
duced  earlier  as  a  reasonable  threshold  for  seriously  considering  the 
use  of  SCE,  and  the  number  of  offerors  shown  in  Figure  3-3,  the  SCE 
cost  is  0.6%  of  the  total  program  cost.  While  using  SCE  to  help  select 
the  most  qualified  offeror  will  not  eliminate  cost  or  schedule  problems,  it 
will  point  out  the  offeror(s)  with  the  best  proven  practices  to  mitigate  such 
problems,  which  can  more  than  pay  for  itself  over  the  life  of  the  contract. 

Suggestions  For  Optimizing  Resources 

Acquisition  organizations  performing  SCEs  routinely  for  multiple  acqui¬ 
sitions  can  optimize  the  resources  required  for  SCE  implementation  by 
assigning  full-time  SCE  support.  This  option  offers  the  greatest  savings 
in  both  cost  and  personnel.  Full-time  support  could  take  the  form  of  ded- 
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icated  personnel  within  the  organization  or  from  a  contracted  organiza¬ 
tion.  Dedicated  support  can  take  on  two  levels  of  involvement. 
Personnel  can: 

•  Help  with  the  source  selection  documentation  needed  to  use 
SCE,  identify  team  members,  and  coordinate  their  training. 

•  Augment  the  SCE  teams  for  specific  acquisitions  by 
participating  in  the  on-site  visits. 

Fully  dedicated  personnel,  who  have  already  gone  up  an  SCE  learning 
curve,  should  be  capable  of  implementing  local  SCE  policies  and  proce¬ 
dures  quickly  and  effectively,  which  should  reduce  overall  costs. 

The  use  of  full-time  resources  to  augment  a  program’s  SCE  team  will  in¬ 
sure  organizational  consistency  in  the  practice  of  the  SCE  Method,  and 
assist  the  training  of  personnel  through  a  form  of  on-the-job  technology 
transition.  Utilizing  at  least  one  full-time  resource  will  act  as  a  significant 
acquisition  “force  multiplier”  when  it  comes  to  implementing  SCE  in  an 
organization. 

The  following  approaches  to  cost  reduction  should  be  avoided  under  all 
circumstances  because  they  would  not  follow  the  SCE  Method. 

•  Team  members  untrained. 

•  Using  Maturity  Questionnaire  responses  alone  without 
performing  site  visits. 

•  Accepting  self  assessment  results  in  lieu  of  conducting  site 
visits. 

These  approaches  undermine  the  consistency,  repeatability,  and  reli¬ 
ability  of  the  SCE  Method. 


3.1 .2  Recipient 

From  the  recipient’s  view,  the  decision  to  use  SCE  is  a  given.  The  recip¬ 
ient  organization  will  be  aware  of  the  acquisition  organization’s  ultimate' 
decision  to  use  SCE.  As  a  result,  the  recipient  must  understand  what  the 
decision  to  use  SCE  actually  means  to  the  organization.  The  U.S.  Gov¬ 
ernment  has  been  steadfast  in  its  support  of  total  quality  management 
principles  and  the  belief  that  concentrating  on  software  process  im¬ 
provement  will  pay  dividends  in  the  form  of  better,  less  costly,  shortened 
time  to  customer/market  software  systems. 
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To  remain  competitive  on  successive  acquisitions,  development  organi¬ 
zations  must  improve  their  software  development  processes.  In  contract 
monitoring,  software  capability  evaluation  can  be  used  to  measure 
progress  relative  to  the  process  capabilities  measured  during  source  se¬ 
lection,  augmenting  an  action  plan  for  improvement.  The  Government’s 
Performance  Risk  Analysis  Group  (PRAG)  can  evaluate  process  im¬ 
provement  based  on  past  performance  risk  assessments  of  the  software 
process. 

By  making  SCE  a  discriminator  in  conducting  acquisitions,  program  of¬ 
fices  will  motivate  contractors  to  focus  on  software  process  capability  as 
a  means  of  retaining  or  increasing  acquisition  agency  business.  Given 
the  premise  that  product  quality  will  follow  process  quality,  focusing  on 
software  process  improvements  resulting  in  increased  process  maturity 


will  increase  the  likelihood  of 

•  accurate  estimates 

•  decreased  variance  among  projects 

•  reduced  cost  and  schedule  targets 

Although  there  is  no  definitive  study  validating  these  benefits  quantita¬ 
tively,  there  is  significant  anecdotal  evidence  from  individual  government 
and  industry  organizations  to  suggest  these  benefits  are  real.  See  the 
listing  of  Case  Studies  of  Software  Process  Improvement  located  in  Ap¬ 
pendix  F  References. 

A  focus  on  improving  software  process  capability  should  lead  to  error 
prevention,  earlier  detection  of  errors  in  the  lifecycle,  and  an  ability  to 
manage  requirements  changes  effectively.  Improvement  in  processes 
that  yield  earlier  detection  of  errors  and  better  management  of  the  re¬ 
quirements  change  process  should  save  the  acquisition  agency  money 
over  the  lifecycle  of  major  systems. 

3.1 .2.1  Development  Organization  Resource  Requirements 

The  costs  of  an  SCE  to  the  development  organization  are  signifi¬ 
cant  ^though  not  always  as  high  as  those  of  the  acquisition  agency/or¬ 
ganization.  Considerable  preparation  time  is  expended  by  a  company; 
the  company  is  typically  trying  to  put  its  best  foot  forward  for  the  acqui- 
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sition  agency,  especially  if  the  SCE  is  done  in  conjunction  with  a  source 
selection.  Thus,  all  development  organizations  will  perform  some  prepa¬ 
ratory  tasks  to  accommodate  an  SCE. 

Table  3-3  provides  an  estimate  in  person-weeks.  The  preparation  time 
of  four  person-weeks  accounts  for  one  person  full-time  for  four  weeks  or 
two  individuals  working  full-time  for  two  weeks  prior  to  the  SCE  site  visit. 
Activities  include  identifying  projects  for  review,  getting  maturity  ques¬ 
tionnaires  and  project  profiles  completed,  and  coordinating  the  site  visit 
activities. 
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Table  3-3:  Example  SCE-Imposed  Development  Organization  Costs 


The  site  visit  costs  are  those  associated  with  conducting  individual  inter¬ 
views  (that  is,  the  cost  of'the  interviewees’  time).  The  final  costs  are 
those  incurred  by  the  offeror  point  of  contact  (POC),  who  supports  the 
SCE  team,  coordinates  activities  with  the  company,  and  schedules  indi¬ 
viduals  for  interviews.  This  POC  also  prepares  individuals  before  their 
interviews  and  debriefs  the  interviewees  after  each  interview.  These 
costs  vary  considerably  from  organization  to  organization. 

Costs  can  increase  if  some  contractor  staff  must  travel  to  another  site  to 
accommodate  an  SCE.  Sometimes  the  projects  selected  for  the  evalua¬ 
tion  are  within  a  product  line  and  division  that  may  be  at  different  loca¬ 
tions.  While  the  SCE  Method  encourages  project  selection  within  the 
same  geographical  location,  this  cannot  always  be  done  because  of  the 
development  organization’s  organizational  structure.  Development  or¬ 
ganization  personnel  traveling  to  accommodate  an  SCE  will  not  only  be 
spending  travel  funds,  their  SCE-associated  labor  costs  will  be  greater 
as  well.  Under  these  circumstances,  the  SCE  team  should  work  with  the 
development  organization’s  POC  to  help  minimize  the  impact  on  those 
affected  projects. 

The  development  organization’s  preparatory  costs  are  significant:  for  a 
period  of  at  least  one  week,  the  offeror’s  operations  will  be  disrupted  by 
SCE  site  activities,  company  SCE  preparation,  and  debriefing  activities. 
These  estimated  costs  will  change  depending  on  the  contractor,  and 
also  as  contractor  familiarity  with  the  SCE  process  grows  and  prepara¬ 
tion  becomes  more  standardized. 
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3.2  Activity  2  Develop  Evaluation  Plan 

Table  3-4  provides  an  overview  of  this  activity. 


Generic  SCE 

Supplier  Selection  SCE  | 

Purpose 

Create  an  artifact  that  will  guide  and  define 
execution  of  the  evaluation  such  that  it  is  in 
concert  with  business  needs  and  constraints. 

Create  a  plan  guiding  the  acquisition-specific 
implementation  of  SCE. 

Actions 

Develop  and  refine  initial  evaluation  plan. 

Develop/provide  inputs  for  documents  such  as 

•  Sources  Sought  Commerce  Business  Daily 
(CBD). 

•  Source  Selection  Plan  (SSP) 

•  Evaluation  Plan  (EP) 

•  Request  for  Proposal  (RFP). 

The  following  usually  occur  during  activities 

2-  4. 

•  definitize  SCE  Role  in  Source  Selection 
(e.g.,  specific  criterion,  general 
consideration). 

•  insert  SCE  language  into  RFP 

Outcome 

The  sponsor  and  team  leader  agree  on  the 
planned  course  of  action  for  the  evaluation. 

The  sponsor  and  team  leader  agree  on  the 
detailed  implementation  of  SCE  in  the 
acquisition. 

Table  3-4:  Develop  Evaluation  Plan  Overview 


3.2.1  Evaluator 

3.2.1 .1  Scheduling  SCE  In  a  Supplier  Selection 

This  section  presents  the  source  selection  process  using  the  RFP  re¬ 
lease  point  as  the  date  from  which  to  build  a  source  selection  schedule 
that  incorporates  SCE.  This  information  is  presented  from  a  government 
perspective  i.e.,  the  government  is  planning  to  use  SCE  teams  in  a  spe¬ 
cific  procurement,  government  and  its  team  are  the  evaluator.  Industry 
users  will  find  the  information  useful,  but  will  need  to  tailor  their  evalua¬ 
tion  planning  to  their  individual  needs.  Although  specific  steps  are  not 
necessary  from  an  individual  contractor  basis,  they  do  provide  a  se¬ 
quencing  of  activity  at  a  level  of  detail  useful  to  any  company  contem¬ 
plating  using  SCE  to  select  a  development  team  or  suppliers  of 
software.Government  contracting  knowledge  of  the  source  selection 
schedule  is  critical  to  the  development  of  the  overall  evaluation  plan.  The 
following  subsections  will  examine  the  SCE  schedule  within  a  source  se¬ 
lection  before  and  after  RFP  release.  Each  will  present  a  schedule  of 
SCE-related  activities  showing  a  range  of  time  in  which  the  activities 
need  to  be  completed,  not  the  time  to  complete  the  activity.  These 
schedules  are  approximate  rather  than  absolute,  and  should  be  tailored 
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to  the  acquisition’s/evaluation’s  circumstances.  Each  activity  on  the 
schedule  is  annotated  with  a  comment  describing  the  activity  and  a  num¬ 
ber,  which  will  be  referenced  in  the  text  for  further  discussion  of  each 
SCE-related  activity. 

SCE  Schedule  Leading  Up  To  RFP  Release 

The  schedule  presented  in  Figure  3-4  refers  to  the  major  SCE-related 
source  selection  activities  that  are  typically  accomplished  before  the  re¬ 
lease  of  the  RFP.  The  first  three  activities — annotated  with  a  “1 “2,”  and 
“3” — show  start  dates  in  the  range  of  seven  or  eight  months  prior  to  the 
release  of  the  RFP.  Depending  on  the  acquisition,  these  dates  could  be 
too  close  or  too  far  from  the  target  RFP  release  date. 

Activity  1 — SCE  Implementation  Planning.  This  is  the  activity  discussed 
in  Activity  1  Analyze  Requirements — deciding  to  use  SCE  in  an  acquisi¬ 
tion.  This  activity  could  continue  until  the  day  the  proposals  are  received, 
depending  upon  the  proposed  application  of  SCE.  Part  of  the  activity 
may  include  inserting  wording  about  planned  SCE  usage  into  the  Com¬ 
merce  Business  Daily  (CBD)  announcement.  This  activity  starts  before 
the  formation  of  an  SCE  team. 

Activities  2  and  3 — Documentation.  This  activity  involves  preparing  the 
documentation  needed  for  the  Source  Selection  Plan  (SSP)  and  the 
RFP.  These  documents  describe  how  SCE  will  be  used  for  the  acquisi¬ 
tion— the  former  for  the  SSA,  SSAC  and  SSEB,  the  latter  for  the  offeror 
community.  j 
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Activity  4 — Pre-Proposai  Conference.  This  is  usually  a  one-day  event  to 
present  the  acquisition  strategy,  contract  type,  evaluation  criteria,  and 
major  program  milestones  to  prospective  offerors.  It  is  an  opportunity  for 
the  offeror  community  to  hear  first-hand  about  the  pending  program  and 
for  the  government  to  receive  feedback  on  the  program  and  their  source 
selection  approach.  Typically,  a  portion  of  the  event  will  be  dedicated  to 
briefing  how  SCE  will  be  used,  and  allowing  offerors  to  seek  further  guid¬ 
ance  or  explanation,  if  needed. 

Activity  5 — Evaluation  Standard  Preparation.  This  activity  is  one  of  the 
most  important  activities  the  SCE  team  leader  or  other  individual  respon¬ 
sible  will  be  engaged  in  related  to  SCE.  The  evaluation  standard  will  dic¬ 
tate  to  the  government  team  how  the  SCE  site  visit  strengths  and 
weaknesses  by  key  process  area  are  measured  and  then  translated  into 
findings  used  in  the  source  selection  decision.  Standards  are  not  provid¬ 
ed  to  the  offerors.  Appendix  B  provides  examples  of  findings  translated 
into  source  selection  data. 
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Activity  6 — SCE  Team  Training  and  Preparation.  This  activity  will  vary  in 
amount  of  work  according  to  the  experience  of  the  team  and  the  SCE  in¬ 
frastructure  in  place  at  the  acquisition  organization  that  supports  the 
team.  It  is  recommended  that  team  training  take  place  within  one  to  two 
months  of  the  actual  site  visits.  If  all  members  of  the  team  have  been 
trained,  but  have  not  worked  together  on  an  SCE,  a  practice  SCE  is  rec¬ 
ommended.  All  team  members  should  have  been  trained  in  SCE  by  the 
SEI  or  one  of  its  licensees. 

SCE  Scheduie  After  REP  Release 

Figure  3-5  on  page  45  depicts  a  typicai  source  selection  schedule  after 
RFP  release.  As  with  previous  schedules,  this  one  is  given  for  illustrative 
purposes  only. 

Activity  1 — Offerors  Prepare  Proposals.  Within  the  acquisition  organiza¬ 
tion,  while  offerors  are  preparing  proposals,  the  month  after  the  RFP  has 
been  released  is  spent  preparing  for  SCE  site  visits.  During  this  period, 
the  SCE  team  should  come  together  to  prepare  for  the  site  visits,  includ¬ 
ing  performing  team-building  activities.  The  offerors  will  have  received 
instructions  in  the  RFP  on  exactly  how  to  prepare  for  the  possibility  of 
SCE  site  visits.  This  will  have  included  specifics  regarding  project  selec¬ 
tion,  responding  to  the  maturity  questionnaire,  and  establishing  a  point 
of  contact  who  will  help  with  the  logistics  of  the  site  visit. 

Activity  2 — Initial  Evaluations.  After  receipt  of  the  proposals,  the  techni¬ 
cal,  cost,  and  past  performance  PRAG  or  other  evaluation  teams  begin 
their  activities.  The  SCE  team  may  also  evaluate  written  proposals  in 
software  area(s).  The  receipt  of  proposals  is  typically  the  initiation  of  the 
formal  preparation  for  on-site  visits  to  the  offerors;  however,  the  visits 
themselves  will  not  be  conducted  until  after  the  competitive  range  deter¬ 
mination  is  made.  The  particular  circumstances  of  the  acquisition  (e.g., 
number  of  offerors,  SCE  team  availability,  competitive  range  determina¬ 
tion)  will  dictate  the  exact  sequence  of  activities  that  occur  for  the  SCE 
team  during  this  period  of  time. 

Activity  3 — SCE  Site  Visits.  The  SCE  team  will  perform  “••site  visits  with 
all  the  offerors  remaining  in  the  competitive  range.  Precisely  when  the 
SCEs  are  to  be  conducted  is  largely  dictated  by  how  SCE  is  being  used 
in  contributing  to  the  source  selection  decision  as  described  in  the  SSP 
or  evaluation  plan.  For  instance,  if  the  SCE  results  will  be  factored  into 
an  item  or  items  of  specific  criteria,  they  should  be  conducted  after  re¬ 
ceipt  of  responses  to  clarification  requests  (CRs)  or  deficiency  reports 
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(DRs)  but  before  face-to-face  discussions.  If  SCE  is  to  be  used  to  sup¬ 
port  PRAG  (past  performance)  findings,  then  site  visits  can  be  accom- 
piished  anytime  after  competitive  range  determination  but  before  best 
and  finai  offers  (BAFOs)  are  issued.  The  instance  described  above 
would  be  considered  as  part  of  the  source  selection  activity,  discussions, 
and  negotiations  described  beiow.  These  two  activities  are  described 
separateiy  for  iilustrative  purposes.  SCE  site  visits  can  be  and  are  con¬ 
ducted  as  separate  events  from  the  formai  legal  definitions  of  discus¬ 
sions  and  negotiations. 

Activity  4— Discussions  /  Negotiations.  This  activity  addresses  that  part 
of  the  process  where  CRs,  DRs,  and  Points  For  Negotiation  (PFNs)  are 
communicated  to  the  offerors  within  the  competitive  range.  These  tools 
can  be  used  by  the  government  to  communicate  SCE  findings  to  the  of¬ 
ferors.  The  government  allows  all  remaining  offerors  to  respond  to  each 
and  then  requests  these  offerors  to  submit  a  BAFO.  The  government  will 
also  begin  developing  “model”  contracts  for  those  offerors  still  within  the 
competitive  range  to  reflect  the  terms  and  conditions  agreed  upon  by 
both  parties  (the  government  and  that  particular  offeror).  Offerors  are 
advised  that  any  deviation  from  the  agreed-to  terms  and  conditions  that 
are  not  traceable  from  their  original  proposal  may  result  in  their  proposal 
being  unacceptable.  This  period,  too,  can  last  more  or  less  time  than  de¬ 
picted  in  Figure  3-5. 
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Activity  5 — BAFO  Evaluations.  Here,  the  government  considers  the  off¬ 
erors’  BAFOs.  This  may  entail  significant  analysis  comparing  the  offer¬ 
or’s  responses  as  to  their  effect  upon  the  analysis  and  determinations 
formulated  to  this  point.  Here  again  the  new  or  revised  information  is  an¬ 
alyzed/evaluated  against  the  approved  evaluation  standards  used  in 
evaluating  the  offerors  initial  proposal. 

Activity  6 — Source  Selection  Decision.  Once  all  of  the  aforementioned 
activities  have  been  completed,  an  award  decision  will  be  made.  The 
SSA  will  have  been  convinced  that  an  equitable,  effective  and  objective 
evaluation  of  each  offeror’s  strengths,  weaknesses,  and  improvement 
activities  has  been  made  by  the  SSEB.  The  time  from  receipt  of  BAFOs 
to  contract  award  can  take  some  time  as  there  are  a  considerable  num¬ 
ber  of  legally  imposed  actions  that  must  take  place  before  announcing 
an  award. 

3.2.1 .2  incorporating  SCE  into  the  Relevant  Acquisition  Documentation 

The  major  documents  related  to  the  source  selection  process  affected 
by  the  incorporation  of  SCE  are  discussed  below.  The  documents  exam¬ 
ined  are:  CBD  announcement,  SSP,  Pre-proposal  Conference  presen¬ 
tation,  RFP,  and  the  Evaluation  Standards.  A  discussion  accompanies 
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each  section  describing  why  a  particular  approach  was  taken  and  con¬ 
tains  at  least  one  example  that  can  be  tailored  to  the  unique  needs  of  the 
acquisition. 

Making  the  Commerce  Business  Daiiy  Announcement 

Figure  3-6  presents  a  slightly  modified  SCE-related  portion  of  an  actual 
CBD  announcement.  This  announcement  informed  the  potential  offerors 
that 


•  SCE  would  be  performed  to  identify  the  offeror’s  software 
process  capability. 

•  an  offeror’s  software  process  capability  would  be  an  integral 
part  of  the  source  selection  decision. 

•  the  government  was  linking  quality,  cost,  and  schedule 
performance  directly  to  software  process  capability. 

•  the  offeror  should  have  an  SPIP  in  place  designed  to  achieve 
higher  maturity  levels  of  the  CMM. 

The  message  from  the  government  is  that  offerors  should  be  implement¬ 
ing  process  improvement  programs  to  achieve  higher  levels  of  process 
capability  and  should  have  a  program  in  place  to  be  a  “viable”  competi¬ 
tor.  The  RFP  that  would  follow  a  CBD  announcement  such  as  that 
shown  In  Figure  3-6  could  reinforce  this  concept  by  requiring  the  submis¬ 
sion  of  a  SPIP  as  an  appendix  to  the  offerors’  proposals. 

The  statement  In  the  (/bd,  “Contractors’  software  process  capability  will 
be  verified  by  analyzing  KPAs  through  the  Defined  maturity  level,  and  a 
demonstrable  softwar'e  process  improvement  program  leading  to  levels 
of  process  capability  higher  than  the  current  capability”  makes  it  clear 
that  the  Defined  maturity  level  is  not  a  contract  requirement.  Rather,  it  is 
a  standard  by  which  the  evaluation  will  be  conducted,  and  the  source  se¬ 
lection  will  consider.  It  essentially  defines  the  "^target  process  capabil¬ 
ity,  which  is  the  capability  the  acquirer  is  seeking  for  this  particular 
acquisition  program. 
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As  part  of  <Acquisition  Agency’s>  overall  effort  to  improve  software  quality,  cost,  and  schedule 
performance,  the  software  process  capability  of  the  responding  offerors  will  be  a  consideration  in  the 
source  selection  decision.  <Acquisition  Agency>  will  use  the  Software  Capability  Evaluation  (SCE) 
method  developed  by  the  Software  Engineering  Institute  (SEI)  to  evaluate  the  current  software 
process  of  responding  offerors  (see  Capability  Maturity  Model  for  Software  (CMU/SEI-93-TR-24, 
February  1 993).  Offerors  who  are  determined  to  be  in  the  competitive  range  will  have  their  software 
process  capability  verified  by  an  SCE  team.  The  SCE  team  will  analyze  key  process  areas  (KPAs) 
through  the  Defined  maturity  level  and  look  for  a  software  process  improvement  program  leading  to 
levels  of  process  capability  higher  than  the  current  capability.  A  conference  will  be  held  at  <time>  on 
<date>  at  <where>  to  answer  any  questions. 


Figure  3-6:  Sample  CBD  Announcement 

Why  place  SCE  wording  into  the  CBD  announcement?  SCE  is  a  tech¬ 
nique  requiring  significant  logistical  planning  and  preparation,  and  not  all 
offerors  will  be  familiar  with  its  use  by  either  the  government  or  compa¬ 
nies  performing  their  own  supplier  selections.  Describing  SCE  in  the 
CBD  allows  the  acquisition  organization  to  provide  a  first  glimpse  of  the 
acquisition  particulars  and  describe  specifics  so  industry  has  a  better  un¬ 
derstanding  of  the  requirements. 


Placing  SCE  in  the  Source  Selection  Plan 

The  SSP  outlines  how  the  source  selection  will  be  conducted  and  sub¬ 
sequently  how  the  award  decision  will  be  made.  This  document  is  not 
seen  by  the  offerors.  SCE  has  a  relatively  small  impact  on  the  production 
of  the  SSP.  SCE  is  discussed  in  several  places  in  an  SSP.  The  following 
subsections  address  several  of  these. 

Source  Screening  |n  this  case,  the  government  would  publish  a  sources-sought  synopsis 
in  the  CBD  requesting  that  interested  offerors  provide  to  the  government 
their  qualifications  in  any  one  of  a  number  of  technical  areas  important 
to  the  acquisition.  The  purpose  of  this  activity  is  to  produce  a  list  of  tech¬ 
nically  qualified  offerors.  Maturity  level  should  not  be  used  as  a  screen¬ 
ing  criterion  at  this  stage.  However,  the  presence  of  an  ongoing  software 
process  improvement  program  could  be  used  as  a  screening  criterion. 

Figure  3-7  provides  sample  wording  to  place  SCE  in  the  Source  Screen¬ 
ing  section  of  the  SSP.  The  hypothetical  source  selection  is  using  Ada 
Software,  Radar  Signal  Processing,  and  Software  Engineering  Capabil¬ 
ity  as  screening  criteria.  The  last  statement  in  this  example  communi- 
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cates  the  organization’s  desire  to  keep  assessment  results  out  of  the 
source  selection  process.  The  SCE  team  should  not  ask  to  see  assess¬ 
ment  results  when  on  site.  Many  organizations’  process  improvement 
programs  can  be  undermined  by  offerors  trying  to  demonstrate  to,  via 
maturity  level  attainment,  the  government  a  process  capability  they  can¬ 
not  support  on  a  new  program. 


^  3.0  Source  Screening 


^  3.1  Screening  Criteria.  Initial  screening  of  potential  offerors  was  conducted  by  the  publication  of  a 
sources-sought  synopsis  in  the  Commerce  Business  Daily  on  <date>.  The  synopsis  requested 
that  interested  offerors  provide  their  qualifications  in  the  following  areas; 

P 

S  a.  Software  Engineering  Capability.  Knowledge  of  software  process  improvement  with  a 
1  verifiable  program  for  process  improvement  using  the  Software  Capability  Evaluation  (SCE) 

I  Method  developed  by  the  Software  Engineering  Institute  (SEI)  to  evaluate  the  current  software 

^  process  of  responding  offerors  (Capability  Maturity  Model  (CMM)  for  Software,  Version 

||  1.1(CMU/SEI-93-TR-24,  Feb  93)  and  Key  Practices  of  the  Capability  Maturity  Model,  Version 

1.1,  CMU/SEI-93-TR-25,  Feb  93)  The  offerors  who  are  determined  to  be  in  the  competitive 
range  after  initial  government  evaluation  of  proposals  is  completed  will  have  their  software 
process  capability  verified  by  an  SCE  team.  The  SCE  team  will  evaluate  key  process  areas 
p  (KPAs)  through  the  Defined  maturity  level  and  look  for  a  demonstrable  software  process 

P  improvement  program  leading  to  levels  of  process  capability  higher  than  the  current  capability. 

^  Do  not  provide  software  process  assessment  results. 


Figure  3-7:  SCE  as  a  Screening  Criterion  in  the  SSP 


SCE  as  a  Specific 
Criterion 


The  following  example  shows  how  to  use  SCE  as  a  specific  criterion. 
Keep  in  mind  that  this  is  only  an  example.  Each  application  of  SCE 
should  be  tailored  to  accommodate  the  unique  demands  of  the  acquisi¬ 
tion. 


Figure  3-8  provides  a  detailed  description  of  how  a  source  selection 
could  use  SCE  in  the  source  selection  evaluation  process. 
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6.4  Evaluation  Criteria 

6.4. 1  Assessment  Criteria 

a.  Soundness  of  approach 

b.  Compliance  with  requirements 


I  6.4.2 


Specific  Criteria 

a.  Technical  Area  -  This  area  will  evaluate  three  items  (Software  Engineering  Capability  (SEC), 
Technical  (TECH)  approach,  and  Management)  represented  here  in  descending  order  of 
importance: 

1 .  Software  Engineering  Capability.  The  government  will  evaluate  the  software  process  by 
reviewing  the  offeror’s  Software  Process  Improvement  Plan  (SPIP)  and  by  using  the 
Software  Engineering  Institute-  (SEI)  developed  Software  Capability  Evaluation  (SCE) 
Method.  The  government  will  determine  the  software  process  capability  by  investigating  the 
Key  Process  Area  (KPAs)  defined  in  the  SEI  Technical  Reports,  Capability  Maturity  Model  for 
Software,  Version  7. 7(CMU/SEI-93-TR-24,  February  1993)  ar\d  Key  Practices  of  the 
Capability  Maturity  Model,  Version  1.1,  (CMU/SEI-93-TR-25,  February  1993).  The  reports 
contain  a  description  of  the  Capability  Maturity  Model  (CMM).The  government  will  perform 
an  SCE  of  each  offeror  remaining  in  the  competitive  range  by  reviewing  current  projects  at 
the  site  proposing  on  this  contract  and  comparing  processes  used  on  those  projects  in  the 
written  proposal/SPIP. 

The  evaluation  will  result  in  a  composite,  substantiated  through  individual  interviews  and 
reviews  of  documentation,  of  the  offeror’s  software  process  on  the  government-selected 
projects.  A  risk  assessment  to  compare  proposed  practices  to  current,  validated  practices 
may  be  performed.  The  evaluation  team  will  determine  findings  of  the  offeror’s  strengths, 
weaknesses,  and  improvement  activities  in  all  KPAs  through  the  Defined  maturity  level. 
Results  of  the  SCE  will  not  be  pass/fail.  Identified  weaknesses  will  be  provided  in  writing 
during  subsequent  discussions.  The  offeror  will  be  allowed  to  respond  to  the  findings  with  a 
limited  number  of  pages  of  text.  The  on-site  evaluators  may  be  separate  and  distinct  from  the 
proposal  evaluation  team  and  may  include  a  government  contracting  representative.  All 
evaluators  have  been  trained  in  the  SCE  Method. 


Figure  3-8:  SCE  as  a  Specific  Criterion  For  The  SSP 


In  Figure  3-8,  the  use  of  SCE  as  a  specific  criterion  falls  under  one  of 
three  items  under  the  technical  area  (SEC,  TECH  Approach,  and  Man¬ 
agement)  in  this  case,  SCE  is  the  most  important  technical  item.  The 
SCE  findings  in  this  case  will  form  the  basis  of  an  item  color  code  and 
risk  assessment  and  will  be  compared  to  an  evaluation  standard  based 
on  the  Defined  maturity  level.  The  SCE  is  not  pass/fail — ^that  is,  being 
less  than  Defined  maturity  level  will  not  exclude  an  offeror  from  the  com¬ 
petitive  range.  In  this  example,  all  offerors  within  the  competitive  range 
will  experience  an  SCE  site  visit  and  be  given  the  opportunity  to  respond 
to  their  evaluated  software  process  weaknesses  through  DRs,  CRs,  and 
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PFNs.  Complete  SCE  findings  (strengths,  etc.)  should  be  provided  to 
each  unsuccessful  offeror  during  the  post-award  debriefings  and  to  the 
winning  contractor  at  the  post-award  conference. 

What  is  presented  in  Figure  3-8  tracks  closely  with  what  is  presented  in 
a  later  section  “Placing  SCE  in  the  Request  for  Proposal.” 

Presenting  SCE  at  the  Pre-Proposal  Conference 

The  pre-proposal  conference  is  an  important  opportunity  for  the  offerors 
to  learn  the  specific,  detailed  requirements  of  the  contract  and  for  the 
government  to  communicate  its  intentions  and  receive  feedback  from 
the  potential  offerors.  The  presentation  consists  of  the  following  parts  to 
guide  the  interaction  with  the  prospective  offerors; 

•  The  activities  that  usually  take  place  in  an  SCE. 

•  The  SCE  team  process. 

•  A  description  of  validation  procedures. 

•  A  sample  of  the  documentation  that  may  be  looked  at  by  the 
SCE  team. 

•  A  sample  site  visit  schedule. 

•  The  CMM  and  KPAs  against  which  the  team  will  evaluate  the 
offerors. 

•  A  sample  of  the  findings  the  SCE  team  will  produce  before 
leaving  the  site. 


Placing  SCE  in  the  RFP 

This  section  contains  the  information  needed  to  incorporate  SCE  into  the 
RFP.  The  RFP  is  the  document  used  by  the  government  to  explain  to  of¬ 
ferors: 

•  The  government’s  requirements. 

•  Evaluation  criteria. 

•  The  mechanisms  that  will  be  employed  in  making  the  source 
selection  decision. 

•  How  to  propose  for  the  contract. 

The  examples  in  this  section  continue  the  approach  of  previous  exam¬ 
ples  of  having  Software  Engineering  Capability  being  used  as  a  specific 
criterion  in  the  Technical  area  of  the  proposal.  If  the  SCE  findings  are 
planned  to  be  used  as  a  consideration  under  performance  risk,  these  ex¬ 
amples  can  be  easily  tailored  to  meet  such  usage. 
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General  Language  for 
the  Request  for 
Proposal 


Regardless  of  how  SCE  is  to  be  used  in  making  the  source  selection  de¬ 
cision,  the  description  of  its  use  is  found  in  Sections  L  (Instructions,  Con¬ 
ditions,  and  Notices  to  Offerors)  and  M  (Evaluation  Factors  for  Award)  of 
the  RFP.  The  example  shown  in  Figure  3-9  closely  mirrors  an  actual  de¬ 
scription  of  SCE  use  found  in  Section  M  of  an  RFP.  The  example  begins 
by  identifying  Software  Engineering  Capability  as  an  item  under  the 
technical  area  (specific  criterion). 


For  this  example,  there  were  two  areas  of  evaluation:  1)  Technical  and 
2)  Cost.  The  specific  reference  to  SCE  in  the  RFP  begins  by  describing 
in  general  terms: 

•  What  will  be  evaluated  in  the  technical  area  (process 
capability)  and  the  importance  placed  on  each. 

•  What  the  technical  basis  of  the  evaluation  is  (the  CMM  KPAs). 

•  What  the  results  of  the  evaluation  will  be  (identify  strengths, 
weaknesses,  and  risk  which  will  also  consider  improvement 
activities  by  KPA). 

•  How  the  government  will  conduct  the  evaluation  (select  the 
projects  to  be  reviewed,  conduct  interviews,  and  review 
documentation). 
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B.  Evaluation  Criteria 

1 .0  Introduction 

2.0  Basis  for  Contract  Award 

3.0  General  Considerations 

4.0  Assessment  Criteria 

4.1  Specific  Criteria 

4.1.1  Technical  Area 

a.  Technical  Area  -  This  area  will  evaluate  three  items  (SEC,  TECH  approach  and 
Management)  represented  here  in  descending  order  of  importance: 

1 .  Software  Engineering  Capability.  The  government  will  evaluate  the  software  process  by 
reviewing  the  offeror’s  Software  Process  Improvement  Plan  (SPIP)  and  by  using  the 
Software  Engineering  Institute  (SEI)-developed  Software  Capability  Evaluation  (SCE).  The 
government  will  determine  the  software  process  capability  by  investigating  the  Key 
Process  Area  (KPAs)  defined  in  the  SEI  Technical  Report,  Capability  Maturity  Model  for 
Software,  Version  f.  r(CMU/SEI-93-TR-24,  February  1993)  and  Key  Practices  of  the 
Capability  Maturity  Model,  Version  1.1,  (CMU/SEI-93-TR-25,  February  1993).  The  reports 
contain  a  description  of  the  Capability  Maturity  Model  (CMM)  and  the  SEI-defined  maturity 
levels.  The  government  will  perform  an  SCE  of  each  offeror  remaining  in  the  competitive 
range  by  reviewing  current  projects  at  the  site  proposing  on  this  contract  and  comparing 
methods/processes  used  on  those  projects  in  the  written  proposal/SPIP. 

The  evaluation  will  result  in  an  organizational  composite,  substantiated  through  individual 
interviews  and  reviews  of  documentation,  of  the  offeror’s  software  process  activities  on  the 
government-selected  projects.  A  risk  assessment  to  compare  proposed  practices  to 
current,  validated  practices  may  be  performed.  The  evaluation  team  will  determine  findings 
of  the  offeror’s  strengths,  weaknesses,  and  improvement  activities  in  all  KPAs  through  the 
Defined  maturity  level.  Results  of  the  SCE  will  not  be  pass/fail.  Identified  weaknesses  will 
be  provided  in  writing  during  subsequent  discussions.  The  offeror  will  be  allowed  to 
respond  to  the  findings  with  a  limited  number  of  pages  of  text.  The  on-site  evaluators  may 
be  separate  and  distinct  from  the  proposal  evaluation  team  and  may  include  a  government 
contracting  representative.  All  evaluators  have  been  trained  in  the  SCE  Method. 

1.2  Cost/Price  Area... 


Figure  3-9:  General  RFP  Language  To  Include  SCE  (Section  M) 
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References  to  Maturity 
Levels  in  the  RFP 


While  Figure  3-9  treats  the  maturity  level  as  a  basis  for  evaluation  rather 
than  a  requirement,  SCE  V3.0  allows  the  sponsor  to  decide  whether  a 
Maturity  Level  Rating  will  be  determined  by  the  SCE  team  provided  nec¬ 
essary  criteria  are  met.  Election  to  determine  a  Maturity  Level  Rating 
should  be  explicit  to  the  offerors. 


Placing  SCE  in  the  Evaluation  Plan 

This  section  provides  a  sample  of  the  type  of  information  needed  to  in¬ 
corporate  SCE  into  the  EP,  and  to  assist  in  the  preparation  of  an  evalu¬ 
ation  standard  for  SCE  findings. 

Evaluation  standards  must  be  completed  before  RFP  release.  Evalua¬ 
tion  standards  are  written  in  a  detailed  manner  to  enhance  the  ability  to 
discriminate  among  the  offerors. 

It  is  imperative  that  the  SCE  team  leader  be  a  member  or  advisor  to  the 
SSEB  to  work  with  SSEB  members  to  apply  the  evaluation  standard  to 
the  findings  from  each  of  the  offerors. 

Sample  Evaluation  The  example  presented  in  Table  3-5  is  a  sample  evaluation  standard  for 

standard  evaluating  strength,  weaknesses  and  improvement  activities  applied  to 

the  reference  Model  (CMM  KPAs).  This  standard  would  be  used  by  the 
SCE  evaluation  teams.  Figure  3-10  provides  an  example  of  a  stream¬ 
lined  evaluation  standard  that  would  reside  as  part  of  the  EP.  This  is  a 
detailed  evaluation  standard  written  describing  the  requirements  for  the 
acquisition.  This  implies  that  if  the  requirement  is  met,  the  standard  is 
met,  and  the  offeror  is  scored,  in  the  case  of  USAF  acquisitions.  Green. 
If  the  standard  is  exceeded,  the  offeror  is  scored  Blue.  If  the  requirement 
is  not  met,  depending  on  how  near  it  is  to  being  met,  the  offeror  is  scored 
as  Yellow.  A  Red  score  denotes  serious  deficiencies  (failure  to  meet  re¬ 
quirements).  The  application  of  the  color  ratings  to  a  standard  should  be 
correlated  with  the  appropriate  regulations  and  procurement  policies  af¬ 
fecting  your  acquisition.  See  Appendix  B,  SCE  Implementation  Exam¬ 
ples,  for  a  full  depiction  of  the  USAF  use  of  colors  with  SCE. 
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Finding  Description 


Strength  A  strength  indicates  a  particular  part  of  the  software  process  capability  that  Is  sufficiently 

robust  to  mitigate  the  development  risks  due  to  the  software  process. 

Weakness  A  weakness  indicates  a  particular  part  of  the  software  process  capability  that  has 
characteristics  that  increase  the  risks  due  to  the  software  process. 

Improvement  A  process  improvement  that  is  not  yet  institutionalized  for  example,  a  pilot  program  that 

Activity  implements  a  new  configuration  management  process.  It  indicates  potential  mitigation 

of  risk  due  to  software  process. 

Table  3-5;  Strength,  Weakness,  and  Improvement  Activity  Guidelines 


BDESCRIPTION:  Software  Engineering  Capability— The  government  will  evaluate  the  offeror's 
■organizational  software  process  capability  by: 

Performing  a  Software  Capability  Evaluation  (SCE). 

M*  Evaluating  the  offeror's  program  for  software  process  improvement. 

P*  Evaluating  the  extent  to  which  the  offeror’s  software  process  supports  the  goals,  objectives,  and 
B  requirements  of  the  solicitation. 

||STANDARD-The  standard  is  met  when  the  offeror  presents  a  sound,  compliant  approach  and: 

||l .  The  findings  from  the  SCE  show  that  the  offeror  Is  strong  or  acceptable  In  each  of  the  following  key 
||  process  areas: 

•  Software  Configuration  Management 
^  •  Software  Quality  Assurance 

||  •  Software  Subcontract  Management 

II  •  Software  Project  Tracking  and  Oversight 
Hi  •  Software  Project  Planning 

S  •  Requirements  Management 

P2.  The  findings  from  the  SCE  show  that  the  offeror  is  strong  or  acceptable  in  at  least  one  of  the  following 
II  software  process  areas: 

^  •  Peer  Reviews 

p  •  Intergroup  Coordination 

B*  Software  Product  Engineering 
•  Integrated  Software  Management 

g*  Training  Program 

•  Organization  Process  Definition 
|i  •  Organization  Process  Focus 

^|3.  The  Software  Process  Improvement  Plan  submitted  with  the  offeror's  proposal  portrays  the  offeror’s 
g  current  process  capability  realistically  and  presents  a  realistic  plan  for  process  improvement.  The 

8  offeror’s  plan  is  consistent  with  the  SCE  findings.  The  SPIP  outlines  the  offeror’s  plan  to  achieve 

higher  maturity  levels  and  demonstrates  that  the  offeror  understands  software  process  Improvement, 
P  both  technically  and  In  the  effort  required  to  increase  and  sustain  higher  maturity  levels. 


Figure  3-10:  Streamlined  SCE  Evaluation  Standard 
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Maturity  Level  Rating  How  well  the  offeror’s  Observed  processes  satisfy  the  reference  model 

(CMM)  is  the  most  important  consideration  the  evaluation  team  must  as¬ 
sess.  Each  Maturity  Level  of  the  CMM  has  an  associated  set  of  KPAs 
and  key  practices  against  which  the  offeror’s  observed  processes  are 
evaluated.  All  collected  data  (questionnaires,  presentations,  documen¬ 
tation  reviews,  and  interviews)  are  compared  to  the  CMM.  Based  upon 
this  comparison,  the  SCE  team  must  determine  the  degree  of  conform¬ 
ance  of  the  offeror’s  processes  with  each  of  the  CMM  KPAs.  The  levei 
of  conformance  to  the  model  and  the  relative  importance  of  the  individ- 
uai  KPAs  to  the  particular  acquisition  are  then  used  to  rate  the  offeror’s 
processes  observed  as  strong,  acceptable  or  weak.  General  guidelines 
for  developing  a  SCE  CMM  rating  are  shown  in  Table  3-6.  Note  that  SCE 
V3.0  now  has  a  rating  baseiine  option.  The  options  defined  allow  for  the 
determination  of  a  Maturity  Level,  the  standard  SCE  approach  of  select¬ 
ing  model  components  for  a  deep  versus  broad  evaluation,  or  selecting 
model  components  for  less  than  full  coverage.  Table  3-6  is  appiicable  to 
the  second,  middle  option  whereby  direct  determination  via  model  cov¬ 
erage  of  a  Maturity  Level  Rating  is  not  feasible.  Note  that  the  application 
of  Table  3-6  assigns  an  adjective  rating  to  the  KPAs  of  a  particular  Ma¬ 
turity  Levei  as  whole.  This  is  not  a  rating  whereby  Maturity  Level  attain¬ 
ment  has  been  verified. 


Rating 

Description  I 

Strong 

Conformance  to  a  majority  of  the  model  criteria  is  strong  and  there  are  no  significant 
areas  of  non-conformance. 

Acceptable 

Conformance  to  a  majority  of  the  model  criteria  is  acceptable  and  there  are  no 
significant  areas  of  non-conformance. 

Weak 

There  are  significant  areas  of  non-conformance  to  the  model  criteria. 

Table  3-6:  Capability  Assessment  Guidelines 


This  section  present  an  exampie  of  an  evaluation  standard  which  de- 
emphasizes  maturity  ievels  whiie  keeping  with  the  spirit  of  the  CMM.  A 
Maturity  Level  Rating,  if  eiected  to  be  done  and  the  necessary  criteria 
met,  would  enable  the  team  to  summarize  the  current  state  of  a  particu¬ 
lar  contractor’s  software  development  processes.  Trained  SCE  users 
are  able  to  take  this  example  and  tailor  it  to  meet  the  specific  needs  of 
their  acquisition.  Thus,  SCE  can  contribute  effectively  to  the  source  se¬ 
lection  decision.  The  findings,  provided  to  the  SSEB  by  the  SCE  team. 
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are  a  snapshot  of  process  capability  for  a  specific  site  at  a  particular 
point  in  time.  The  way  those  findings  are  used  by  the  acquisition  organi¬ 
zation  can  be  modified  through  the  design  of  the  evaluation  standard. 


3.2.2  Recipient 

The  CBD  announcement  depicted  in  Figure  3-6  on  page  47  will  probably 
be  the  first  inkling  to  a  development  organization  that  the  government  is 
contemplating  the  use  of  SCE  in  a  formal  acquisition.  As  described,  this 
notice  gives  the  recipient  development  organization  the  opportunity  to 
decide  whether  to  bid  or  not  bid  the  particular  solicitation.  In  the  situation 
of  the  more  informal  industry  organization  to  organization  solicitation, 
this  may  take  the  form  of  a  letter  or  memorandum. 

With  a  decision  to  bid  or  compete;  the  development  organization  can  ini¬ 
tiate  appropriate  arrangements  regarding  the  following  items  for  prepar¬ 
ing  for  and  subsequently  participating  in  the  execution  of  an  SCE: 

•  Geographical  site  and  candidate  project  submissions 

•  Conduct  internal  SCE  to  assess  candidate  site  and  projects’ 
readiness  for  formal  SCE 

•  Documentation  assembly 

•  Traceability  matrices  (e.g.,  CMM  components  to  company 
documents) 

•  Acquisition  schedule  impacts 

•  Project  perionnel  availability 

•  Facilities  preparation 
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3.3  Activity  3  Select  and  Prepare  Team 

Table  3-7  provides  an  overview  of  this  activity. 

1  Generic  SCE 

Supplier  Selection  SCE  I 

Purpose 

Have  an  experienced,  trained  team  ready  to 
execute  the  evaluation. 

Have  an  experienced,  trained  team  ready  to 
execute  the  acquisition  SCE. 

Actions 

Select  and  prepare  the  SCE  team. 

Select  and  prepare  SCE  personnel,  with 
acquisition  requirements  in  context 

Continue  to  prepare  the  SSR  ER  and  RFR 

Outcome 

An  experienced,  trained  team  is  ready  to 
execute  the  evaluation  process. 

An  experienced,  trained  SCE  team  is  ready  to 
execute  the  acquisition  SCE. 

Table  3-7:  Select  and  Prepare  Team  Overview 


3.3.1  Evaluator 

3.3.1 .1  Selecting  the  SCE  Team 

Selecting  qualified,  experienced  software  acquisition  personnel  to  serve 
as  SCE  team  members  is  one  of  the  most  difficult  and  important  tasks  a 
program  or  software  manager  may  do  in  the  course  of  implementing 
SCE  in  an  acquisition.  The  key  considerations  for  assembling  an  SCE 
team  are: 

•  Individual  experience. 

•  Team  skills  and  experience. 

•  Individual  interpersonal  skills. 

•  Team  leadership  skills,  team  building  activities,  and  team 
staffing. 

Individual  Experience 

SCE  team  members  should  be  selected  from  the  most  experienced  peo¬ 
ple  available.  The  most  successful  teams  will  be  those  that  average  at 
least  ten  years  of  appropriate  development  or  management  (acquisition) 
experience  across  the  team.  At  least  one  member  of  the  team  should 
have  source  selection  experience.  (In  the  case  of  commercial  applica¬ 
tions,  one  team  member  should  be  experienced  in  the  company’s  poli¬ 
cies  and  procedures  for  selecting  suppliers,  subcontractors  or  teaming 
partners.)This  is  important  because  what  can  and  cannot  be  done  during 
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a  source  selection  is  different  from  what  is  permissible  after  award.  An 
acceptabie  spread  of  experience  levels  that  have  been  found  to  be  suc¬ 
cessful  in  an  SCE  team  is 

•  At  least  one  or  two  senior  personnel  (more  than  ten  years 
appropriate  experience). 

•  Two  or  three  mid-level  personnel  (eight  to  ten  years 
appropriate  experience)  and  ideally,  participation  in  at  least 
one  SCE  as  team  members. 

•  One  junior  person  (five  years  appropriate  experience).  Note: 
This  is  a  recommended  minimum.  Junior  personnel  typically 
will  not  have  the  background  to  understand  certain  aspects  of 
software  processes  they  will  observe.  No  team  member 
should  have  less  than  five  years  of  professional  experience. 

Team  Skills  and  Experience 

The  background  of  SCE  team  members  should  strike  a  balance  among 
software  technical,  software  management,  and  software  acquisition  ex¬ 
perience.  They  should,  as  a  minimum,  share  a  mix  of  knowledge  and  ex¬ 
perience  in  the  following  disciplines; 


•  Acquisition  policies  and  procedures  (particularly  source 
selection  procedures  or  company  specific  acquisition  policy 
and  procedures) 


•  Project  management  and  planning 

•  Software  configuration  management 

•  Software  design,  development,  and  methodologies 

•  Software  quality  assurance 

•  Systems  engineering 

•  Technical  requirements  of  the  contract 

•  Testing 

•  Application  domain,  e.g..  Avionics,  Missiles,  C3I,  MIS, 
databases. 


In  general,  expertise  will  be  necessary  from  at  least  one  team  member 
in  each  of  the  key  process  areas  to  be  investigated.  Certain  areas  may 
be  stressed  over  others  depending  on  the  acquisition. 
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Interpersonal  Skills 

SCE  team  members  must  be  “team  players.”  Conducting  SCEs  requires 
professional  judgement  and  team  consensus — attributes  that  are  neces¬ 
sary  to  work  effectively  in  an  SCE  team  are  patience  and  perseverance. 
Past  experience  has  demonstrated  that  if  team  members  lack  interper¬ 
sonal  skills— essential  to  fostering  good,  open  communications  between 
team  members  and  the  contractors — ^the  team  is  less  effective,  less 
credible,  and  less  motivated  to  fulfill  the  SCE  objectives.  The  ability  to 
communicate  with  the  contractor  and  other  team  members  is  the  es¬ 
sence  of  SCE  teamwork. 

Team  Leadership  Skills 

Experience  shows  that  an  effective  team  leader  is  critical  to  the  opera¬ 
tion  of  the  team.  The  team  leader: 

•  Ensures  that  the  team  meets  its  schedule  and  objectives, 
encourages  teamwork  and  consensus  building. 

•  Is  the  SCE  team  focal  point  for  both  the  acquisition  office  and 
the  contractors  (planning,  scheduling,  communicating). 

•  Should  have  enough  basic  leadership  skills  to  ensure  that  the 
team  functions  effectively. 

The  team  leader  should  be  the  most  qualified,  based  on  knowledge,  ex¬ 
perience,  and  amount  of  direct  SCE  site  visit  experience.  Occasionally, 
teams  can  break  down  when  an  inappropriate  team  leader  is  appointed. 
Program  office  management  should  be  on  guard  to  prevent  this  from 
happening.  Previous  SCE  experience  should  be  a  criterion  for  assign¬ 
ment  as  an  SCE  team  leader.  New  SCE  team  leaders  should  have  par¬ 
ticipated  on  at  least  two  SCEs  as  a  team  member  before  assuming  a 
leadership  role.  This  experience  of  participating  on  SCEs  will  prepare 
the  new  leader  to  understand  the  SCE  team  process,  team  dynamics, 
and  contractor  sensitivities. 

Team-Building  Activities 

An  essential  aspect  of  preparing  a  team  to  conduct  an  SCE  is  performing 
team-building  activities  before  going  on  site.  Assume  the  SCE  team  has 
never  worked  together:  an  activity  that  would  help  bring  the  individual 
members  together  as  a  team  could  be  an  exercise  whereby  a  simple 
task  is  assigned  and  discussed.  For  example,  each  team  member  would 
interview  the  person  to  his  or  her  right  at  a  table  or  in  a  room.  The  task 
of  the  interviewer  is  to  learn  the  person’s  background,  interests,  and 
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area  of  expertise.  Each  team  member  would  then  introduce  and  briefly 
state  the  results  of  their  interview.  The  team  could  then  identify  its  rela¬ 
tive  background  expertise  areas  to  the  evaluation  task  they  are  being 
asked  to  perform.  For  reference,  Appendix  E  of  this  guide  contains  three 
references,  [Bucholz  87],  [Denton  89],  [Kelly  91],  that  contain  more  on 
teams  and  team-building  activities.  These  exercises  will  help  determine 
how  the  team  members  work  together.  Often,  many  months  may  pass 
after  teams  have  completed  SCE  team  training  and  before  they  conduct 
site  visits.  The  team  building  activities  are  important  for  the  team  mem¬ 
bers  to  re-acquaint  themselves  as  a  single  operating  entity  able  to  reach 
consensus  effectively.  There  may  be  times  when  trained  individuals  are 
brought  together  who  have  not  completed  training  together.  In  this  sce¬ 
nario,  team  building  is  crucial,  since  they  have  not  operated  as  a  team 
before. 

The  success  of  the  SCE  team  hinges  on  its  ability  to  identify  and  reach 
consensus  on  the  strengths  and  weaknesses  of  a  contractor’s  observed 
processes.  An  SCE  team  is  neither  an  autocracy,  where  the  leader  dic¬ 
tates  what  decisions  are  made,  nor  a  democracy,  where  the  team  votes 
and  the  majority  prevails.  Instead,  the  decisions  are  reached  by  team 
consensus,  meaning  all  members  agree  to  the  findings,  and  there  is  no 
significant  minority  dissent  on  issues.  If  consensus  on  an  issue  cannot 
be  reached,  then  there  is  no  finding  in  that  area.  This  is  where  team¬ 
building  activities  will  pay  large  dividends. 


Team  Staffing 

Staffing  the  team  is  another  issue  for  consideration.  Creating  a  “software 
center  of  excellence”  is  an  excellent  mechanism  for  building  a  core  of  in¬ 
dividuals  who  are  highly  skilled  in  conducting  SCE. 

Program  Offices  or  companies  (development  organizations)  performing 
routine  software  process  improvement  will  normally  draw  SCE-trained 
personnel  from  within  their  own  organization.  If  this  pool  does  not  have 
enough  individuals,  a  request  to  the  central  organization  would  then  be 
made  to  assist  in  identifying  team  members.  In  this  manner,  the  program 
offices  or  development  organizations  can  take  advantage  of  key  compo¬ 
nents  mentioned  above  under  individual  and  team  skills.  This  alternative 
makes  better  use  of  limited  software  skilled  resources  while  ensuring 
that  the  program  office  acquisition  expertise,  knowledge  of  the  product 
type  and  contractor  base,  and  “domain”  knowledge  of  the  product  to  be 
acquired  is  present  on  the  team.  Furthermore,  the  “in  house”  resources 
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will  become  valuable  assets  to  the  organization  as  they  gain  more  expe¬ 
rience  conducting  evaluations  for  multiple  supplier  selections  in  different 
programs. 

3.3.1 .2  Training  the  SCE  Team 

Training,  preparation,  and  experience  separates  good  SCE  teams  from 
poor  ones.  There  are  three  levels  of  training  needed  before  an  individual 
should  be  considered  fully  qualified  to  conduct  SCEs: 

•  Preparation 

•  Coursework 

•  Experience 

Preparation 

The  preparatory  stage  consists  of  on-the-job  experience,  prior  training, 
and  professional  reading.  It  is  recommended  that  SCE  team  members 
take  courses  in  team  work,  assertiveness  training,  and  total  quality  man¬ 
agement.  Watts  Humphrey’s  book.  Managing  the  Software  Process 
[Humphrey  89]  and  the  latest  release  of  the  CMM  [Paulk  93a]  are  two 
essential  readings  that  are  provided  to  participants  of  the  evaluation 
team  training  course.  Participation  in  the  one-day  Software  Process  Im¬ 
provement  Overview  Seminar  or  SCE  Refresher  Training  are  also  ben¬ 
eficial  background  to  prepare  team  members. 


Course  Work 

SCE  team  training  consists  of  a  multi-day,  case-study-oriented  course 
that  all  SCE  team  members  must  successfully  complete.  This  course  is 
intended  for  experienced  personnel  who  have  been  selected  to  exercise 
the  SCE  Method.  It  provides  the  knowledge  and  reinforces  the  skills  re¬ 
quired  to  conduct  SCEs  effectively,  and  helps  the  group  develop  into  a 
cohesive  team.  A  sampling  of  course  content  follows; 

•  Team-building  exercises 

•  Preparing  for  the  site  visit 

•  Conducting  interviews 

•  Reviewing  documents 

•  Validating  observations 

•  Developing  and  presenting  findings 
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SCE  teams  need  effective  communicators  willing  to  take  on  different 
roles  (e.g.,  facilitator,  recorder,  interviewer,  timekeeper),  as  demanded 
by  the  dynamics  of  the  team  and  constraints  of  the  acquisition.  The  SCE 
team  needs  to  know  how  to  evaluate  the  contractor  in  relation  to  the 
CMM,  so  a  working  understanding  of  the  CMM  is  required.  Teams  are 
taught  that  processes  implemented  are  to  a  large  degree  dependent  on 
several  non-process  variables  such  as  management  commitment,  and 
the  ability  to  perform  the  given  tasks.  It  takes  experience  to  understand 
these  relationships  and  exercise  professional  judgement,  which  is  why 
the  team  characteristics  and  profile  are  crucial  in  addition  to  the  course- 
work.  Roles  taken  on  by  team  members  to  accomplish  the  site  visit  in¬ 
clude  the  following: 

•  Team  Leader:  manages  process,  keeps  to  agenda, 
guarantees  deadlines  and  deliverables 

•  Facilitator:  sustains  team  spirit,  moves  team  toward 
consensus,  and  encourages  participation 

•  Recorder:  captures  information,  does  not  editorialize,  and  lists 
documents  to  be  reviewed 

•  Reference  Model  Component  Monitor  (e.g.,  CMM  KPA 
Monitor) 

•  Participant:  assists  other  team  members  and  carries  out 
assigned  tasks 

•  Timekeeper  (monitors  elapsed  time,  particularly  during 
interviews)  / 


Experience 

Every  graduate  of  the  SCE  training  program  should  be  a  member  rather 
than  a  leader  of  an  SCE  team  for  at  least  two  SCEs.  Junior-  and  mid¬ 
level  personnel  should  take  part  in  at  least  three  SCEs  before  being  con¬ 
sidered  for  the  team  leadership  role.  Resource  demands  and  time  con¬ 
straints,  however,  may  prevent  this  from  happening.  When  working 
under  such  constraints,  it  is  recommended  that  the  program  office  send 
the  team  to  practice  an  SCE  on  a  volunteer  organizational  office  before 
beginning  the  source  selection.  One  acquisition  team  practiced  doing 
SCEs  on  at  least  three  occasions  to  insure  personnel  were  highly  trained 
for  the  source  selection. 

The  common  denominators  in  discussions  with  individuals  returning 
from  their  first  SCE  is  their  desire  for  more  team  training,  preparation, 
and  time  to  conduct  the  interviews.  SCE  activities  are  not  radically  differ¬ 
ent  from  those  done  in  the  program  office  on  a  day-to-day  basis.  Taken 
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together,  however,  they  are  group  activities  requiring  significant  practice 
and  preparation.  Practicing  as  a  group  will  reveal  individuals’  strengths 
and  weaknesses,  depth  of  required  preparation,  and  how  to  manage  the 
SCE  process  to  capitalize  on  the  team’s  strengths  so  that  more  effective 
and  timely  SCEs  are  conducted. 


3.3.2  Recipient 

Analogous  to  the  considerations  and  activities  that  are  part  of  the  evalu¬ 
ation  team’s  selection  and  preparation,  the  development  organization 
should  take  stock  of  the  personnel  the  SCE  team  is  likely  to  interview 
and  interact  with  during  the  onsite  period.  In  like  manner  the  develop¬ 
ment  organization  should  consider  the  background  and  degree  of  in¬ 
volved  personnel’s: 

•  software  experience 

•  interpersonal  skills 

•  SCE  and  CMM  knowledge 

•  policies,  procedures,  artifact  familiarity 

The  development  organization  similarly  must  assess  the  degree  of  sup¬ 
port  it  will  provide  its  point  of  contact  for  the  SCE.  This  includes  admin¬ 
istrative,  financial,  and  managerial.  This  is  particularly  crucial  where 
more  than  one  geographical  site,  teaming  partners,  and  or  subcontrac¬ 
tors  are  involved.  Programmatic  issues  such  as  which  geographical  site 
the  SCE  team  will  actually  visit,  how  the  other  geographically  separated 
project  sites,  the  teaming  partner  or  subcontractor  are  to  provide  docu¬ 
mentation  for  the  SCE  team,  and  security  for  provided  documentation 
and  data,  disruption  of  ongoing  projects  etc.  must  be  accounted  for. 
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3.4  Activity  4  Obtain  Organization  Information 

Table  3-8  provides  an  overview  of  this  activity. 


1  Generic  SCE 

1  Supplier  Selection  SCE  I 

Purpose 

Obtain  information  from  the  organization  that 
allows  the  team  to  refine  the  plan  and  focus  the 
investigation  in  subsequent  activities. 

Obtain  information  from  the  organization  that 
allows  the  team  to  refine  the  plan  and  focus  the 
Investigation  in  concert  with  the  acquisition 
activities. 

Actions 

RFP  issued. 

SCE  specific  information  is  requested  and 
delineated  via  RFP. 

Outcome 

Development  organization  Information  is 
obtained  by  which  the  team  can  finalize  its 
evaluation  focus,  priorities,  and  logistics. 

Same  as  generic  SCE. 

Table  3-8:  Obtain  Organization  Information  Overview 


3.4.1  Evaluator 

This  is  the  third  of  four  activities  that  make  up  the  General  Preparation 
major  activity  grouping.  With  Activity  4,  obtaining  organizational  informa¬ 
tion  begins  to  define  the  field  of  development  organizations  bidding  or  of¬ 
fering  to  execute  the  contract  that  is  being  solicited.  The  solicitation  or 
RFP  defines  the  field  of  action.  The  respondents  to  the  solicitation  be¬ 
come  the  players,  and  it  is  the  solicitation  sponsor’s  responsibility  to 
maintain  a  fair  and  eguitable  evaluation  or  “level  playing  field.”  The  dis¬ 
cussion  that  follows  provides  typical,  generic  guidance  and  examples  of 
a  government  acquisition  agency’s  communication  with  the  contractor 
community  through  the  RFP. 

3.4.1 .1  Information  Requested 

The  term  “offeror”  is  used  to  denote  organizations  that  are  bidding  on  the 
solicitation.  In  the  instance  of  a  development  organization  using  SCE  to 
select,  team,  or  subcontract  an  analogous  term  such  as  bidder,  respon¬ 
dent  (e.g.,  to  memorandum  for  services  sought)  etc.  could  equally  be 
substituted.  The  context  of  the  discussion  is  unchanged.  This  SCE 
method  activity  remains  the  same  regardless  of  the  actual  participants. 
The  degree  of  formality  with  which  the  information  is  requested  and  sub¬ 
mitted  differentiates  the  application. 
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The  following  proposed  project  information  may  be  required: 

•  '^Maturity  questionnaire 

•  Proposed  "^product  profile 

•  Annotated  organizational  chart  (including  relationship  of 
proposed  product  project  to  the  organization  and  also  projects 
which  show  evidence  of  use;  including  description  of 
organizational  responsibilities) 

•  Draft  SDP 

Evidence  of  use  matrix  (mapping  of  documentation  against  reference 
model  (e.g.,  CMM)  and  against  proposed  process  documentation). 

Requests  for  organizational  Information  is  in  the  form  of  the  RFP  in 
source  selection,  and  is  constrained  by  those  acquisition  rules. 

3.4.1 .2  Incorporating  SCE-Related  Items  into  the  Instructions  for  Proposal  Prepara¬ 
tion  (IFPP) 

The  Instructions  for  Proposal  Preparation  (IFPP,  Section  L)  portion  of 
the  RFP  informs  the  offerors  how  to  prepare  their  proposals  and  comply 
with  the  source  selection  process.  The  portion  of  the  IFPP  that  address¬ 
es  SCE  is  divided  into  five  distinct  pieces: 

•  Organizing  the  SCE-related  information  into  a  separate 
appendix. 

•  Completing  the  Maturity  Questionnaire  Recording  Forms 
(Figure  3-11). 

•  Completing  the  Product  Profiles  (Figure  3-12). 

•  Submitting  the  SPIP  Figure  3-1 3. 

•  General  SCE  instructions  (Figure  3-14). 

Organizing  SCE-Reiated  Information  into  an  Appendix 

Each  acquisition  must  determine  the  best  way  for  their  program  to  in¬ 
struct  offerors  to  organize  their  proposals.  The  goal  is  to  ease  proposal 
evaluations  and  obtain  concise  information  wanted,  and  not  to  impose  a 
burden  on  the  offerors.  Work  with  the  government  PCO  or  the  organiza¬ 
tion’s  contracts/legal  division  to  determine  what  is  best  for  the  acquisi¬ 
tion  and  program.  If  it  is  desired  that  the  SCE  information  be  excluded 
from  a  technical  page  limit,  offerors  could  be  directed  to  place  this  infor¬ 
mation  in  a  separate  appendix. 
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Completing  Maturity  Questionnaire  Recording  Forms 

One  of  the  more  important  SCE-related  items  in  the  IFPP  is  the 
language  shown  in  Figure  3-1 1  describing  how  the  Maturity  Ques¬ 
tionnaire  Recording  Forms  are  to  be  completed.  The  offeror  is  told 
to  select  eight  ongoing  development  efforts  that  best  correlate  to 
the  future  work  under  the  government’s  proposed  contract,  using 
characteristics  such  as  application  domain,  software  size,  devel¬ 
opment  language,  etc.  to  make  the  best  matches. 


3.1  General  Instructions  The  Technical  Proposal  shall  consist  of  the  Executive  Summary  and 
<n>  separate  sections... 

3.2  Executive  Summary... 

3.3  Specific  Instructions... 

3.3.1  Section  I  -  Software  Engineering  Capability  The  offeror  will  provide  the  following 

Information  to  assist  the  government’s  preparation  for  the  Software  Capability  Evaluation  of 
each  offeror: 

a.  The  offeror  will  complete  the  SEI  Maturity  Questionnaire  (MQ)  (current  version)  using  the 
Recording  Form  for  eight  current  projects  at  the  site  proposing  on  this  solicitation  (a 
project  Is  acceptable  only  if  it  has  been  completed  in  the  last  year).  The  offeror  should 
select  those  projects  that  best  match  the  engineering  requirements  of  this  contract.  For 
offerors  with  fewer  than  eight  current  projects  at  the  proposing  site,  submit  MQ  responses 
for  as  many  projects  as  are  available.  For  each  “yes”  response,  please  note  on  the 
comment  line  the  mechanism  or  documentation  justifying  the  response.  The  MQ  can  be 
found  in  Atch  XXX  of  the  IFPP.  The  completed  Recording  Forms  will  be  submitted  with  the 
proposal.  For  all  responses,  please  note  at  the  start  of  the  comment  line  the  degree  of 
implementation  of  each  practice  using  a  letter  identifier  from  the  following  legend: 

A  •  Not  implemented  at  this  time. 

B  -  Not  implemented  at  this  time,  but  desired. 

C  -  Currently  planning  to  implement.  See  improvement  plan. 

D  -  In  the  process  of  implementing. 

E  -  Implemented  with  less  than  a  year’s  experience. 

F  -  Implemented  on  a  project-by-project  basis. 

G  -  Implemented  organizationally. 

H  -  Not  appropriate  for  our  organization. 


Figure  3-1 1 :  Instructions  For  Completing  SEI 
Maturity  Questionnaire 
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Using  the  legend  in  Figure  3-1 1 ,  the  offeror  must  characterize  the  state 
of  institutionalization  of  each  practice.  To  verify  each  response,  the  off¬ 
eror  must  cite  documentation  that  defines  each  practice  it  has  character¬ 
ized.  By  getting  this  information  from  offerors,  the  SCE  team  will  know 
more  about  what  to  look  for  to  verify  a  particular  software  practice,  and 
the  offeror’s  view  of  the  extent  to  which  a  practice  is  implemented;  this 
will  help  focus  on-site  activities  (e.g.,  interviews). 


The  SCE  team  wants  to  identify  and  separate  software  practices  that  are 
institutionalized  (implemented  organizationally)  from  those  that  are  im¬ 
plemented  only  on  a  single  project.  The  questionnaire  should  be  at¬ 
tached  to  the  RFP  so  that  there  is  no  confusion  over  which  questionnaire 
the  offeror  is  required  to  complete  {A  Method  for  Assessing  the  Software 
Engineering  Capabiiity  of  Contractors  [Humphrey  87b]  or  the  Software 
Process  Maturity  Questionnaire,  Capabiiity  Maturity  Model,  version  1. 1 
(April  1994). 


Instructing  Offerors  to 
Complete  Product 
Profiles 


Figure  3-12  directs  the  offeror  to  complete  a  Product  Profile  for  each  of 
the  projects  selected  on  the  Recording  Form.  The  Product  Profile  pro¬ 
vides  information  such  as:  size  of  the  organization,  application  domain, 
product  type,  development  team  approach,  development  language(s), 
type  of  system,  location  of  development,  major  subcontractors,  and  the 
program’s  current  phase(s)  of  development.  See  [Byrnes  96]  for  a  com¬ 
plete  description  of  the  product  profile.This  information  helps  the  govern¬ 
ment  to  select  projects  for  evaluation.  A  Product  Profile  template  should 
also  be  included  as  an  attachment  to  the  IFPP. 


3!3T(conlinued) 


b.  For  the  proposed  effort  and  for  each  project,  the  offeror  will  complete  a  Product  Profile, 
attach  it  to  the  respective  Assessment  Recording  Form,  and  submit  it  with  the  proposal. 
The  Product  Profile  template  can  also  be  found  in  Atch  XXX  of  the  IFPP.  This  document 
shall  be  no  greater  than  one  page  per  project  and  one  page  for  the  proposed  effort. 


Figure  3-12:  instructions  For  Completing  Product  Profiles 
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Submitting  the  Software  Process  Improvement  Pian 

Figure  3-13  addresses  the  SPIP  the  offerors  submit  with  their  proposais. 
In  the  example  provided,  the  offeror  could  not  exceed  15  pages  of  text. 
This  was  an  arbitrary  limit  intended  to  minimize  the  effort  required  by  the 
offerors  and  the  government  during  the  source  selection  process.  The 
government  did  not  want  the  offerors  to  develop  an  SPIP  for  the  acqui¬ 
sition;  rather,  they  wanted  to  see  plans  that  were  already  in  place. 


3.31  (continued) 


c.  The  offeror  will  submit  the  proposing  site’s  Software  Process  Improvement  Plan  (SPIP), 
in  the  format  of  their  choosing,  with  the  proposal.  The  document  will  be  no  more  than  1 5 
pages.  The  SPIP  should  communicate  the  offeror’s  current  software  process  capability  as 
well  as  their  desired  maturity  level,  specific  planned  improvements,  dedicated  resources, 
effort  estimates,  and  a  time  phasing  of  those  improvements  to  bring  the  offeror’s  software 
process  capability  to  the  organization’s  desired  maturity  level. 


Figure  3--13:  Instructions  For  Submitting  the  Software  Process 
Improvement  Plan  (SPIP) 


Generai  SCE  Instructions 

The  last  set  of  instructions  for  the  IFPP,  found  in  Figure  3-1 4,  informs  the 
offeror  of  various  SCE-related  details  that  will  facilitate  a  smoothly  run 
SCE  with  minimal  impact  on  the  offeror’s  organization. 

An  Offeror  Point  of  Contact  is  needed  so  that  the  SCE  team  leader  may 
coordinate  all  activities,  both  before  and  during  the  SCE.  Note  that  the 
offeror  will  be  notified  ten  working  days  in  advance  of  the  site  visit  which 
projects  will  be  evaluated.  There  are  two  reasons  for  this.  First,  this  will 
give  all  the  offerors  the  same  number  of  days  to  prepare  for  the  SCE. 
Second,  because  many  organizations  go  to  great  lengths  to  prepare  for 
an  SCE,  giving  ten  working  days’  notice  will  limit  them  from  expending 
valuable  resources  and  time  on  activities  that  will  have  little  or  no  impact 
on  the  SCE  findings. 
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3.31  (continued) 

d.  After  the  proposal  is  received,  the  government  will  coordinate  a  site  visit  with  those 
offerors  remaining  in  the  competitive  range  to  conduct  the  Software  Capability  Evaluation 
(SCE)  at  the  offeror’s  location.  The  offeror  will  provide,  with  your  proposal,  a  point  of 
contact  and  phone  number  at  the  offeror’s  site  for  the  SCE  team  leader  to  coordinate  all 
SCE  activities.  The  government  will  also  communicate  details  about  the  site  visit  during 
the  coordination  process.  The  offeror  will  be  notified  of  the  projects  to  be  examined 
approximately  ten  working  days  prior  to  the  site  visit.  The  site  visit  dates  selected  by  the 
government  are  not  open  for  discussion. 

e.  If  a  site  visit  is  conducted  with  your  firm,  the  SCE  team  will  need  a  closed  meeting  room 
capable  of  accommodating  at  least  eight  people.  The  offeror  should  have  a  copy  of  the 
organization’s  software  standards,  procedures  and/or  operating  instructions,  and 
organizational  charts  for  the  projects  being  reviewed  in  the  meeting  room  when  the  SCE 
team  arrives.  All  Interviews  conducted  as  part  of  the  SCE  will  be  done  in  private,  one 
individual  at  a  time. 

f.  The  Assessment  Recording  Forms,  Product  Profiles,  and  Software  Process  Improvement 
Plans  will  not  be  included  in  the  page  count  limitations  for  the  proposal. 


Figure  3-14:  Instructions  For  Site  Visit  Coordination 


Facilities  and  The  items  needed  by  the  team  at  the  site  visit  are  mentioned  in  this  sec- 

information  (Figure  3-14).This  information  needs  to  be  provided  here  to  set  ex¬ 

pectations  and  etisure  that  the  offeror  is  reasonably  well  prepared.  The 
SCE  team  must  maintain  the  confidentiality  of  interviews  or  the  entire 
SCE  process  codid  be  undermined.  AN  data  collected  during  the  site  visit 
will  become  part  of  the  source  selection  file  and  will  be  maintained  on  all 
offerors  until  the  contract  is  closed  out. 


Offeror  Exit  Briefing  The  PCO  will  be  the  final  arbiter  in  determining  how  the  findings  will  be 

provided  to  the  offerors.  However,  any  outbriefing  must  advise  the  offer¬ 
or  that  this  may  not  completely  resolve  all  issues  regarding  the  SCE.  It 
is  important  for  all  the  offerors  to  realize  that  they  have  the  right  to  and 
must  specifically  request  a  debriefing  of  the  SCE  findings.  Debriefing  the 
findings  achieves  two  important  goals.  First,  in  a  Total  Quality  Manage¬ 
ment  (TQM)  approach,  the  government  desires  buy-in  from  the  offerors 
regarding  the  results,  and  is  seeking  to  motivate  the  offerors  to  improve 
their  capability.  Second,  the  government  has  the  opportunity  for  direct 
feedback  regarding  the  conduct  of  the  SCE  from  the  offeror’s  perspec¬ 
tive.  This  feedback  can  be  used  to  refine  the  procedures  and  instructions 
for  future  acquisitions. 
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Page  Limitations 


3.4.2  Recipient 


Third-Party  SCEs 


In  most  RFPs,  there  is  a  limit  to  the  number  of  pages  an  offeror  may  use 
in  the  preparation  of  their  proposal.  The  example  provided  here  had 
such  a  requirement.  Consequently,  when  the  IFPP  required  submittal  of 
Assessment  Recording  Forms,  Product  Profiles,  and  an  SPIP,  these 
document  pages  were  excluded  from  the  proposal  page  count  to  ensure 
they  did  not  detract  from  the  technical  content  of  the  proposal  subject  to 
the  page  limitations.  This  is  an  administrative  detail  that  will  allow  page 
counts  to  be  focused  on  the  technical  approach. 

This  section  presented  the  essential  elements  needed  to  accommodate 
SCE  in  an  RFP.  These  references  should  be  tailored  by  the  organization 
to  meet  the  specific  needs  of  the  acquisition.  The  examples  in  the  figures 
presented  can  be  changed  to  accommodate  the  usage  of  the  SCE  find¬ 
ings  as  a  consideration  under  performance  risk  or  a  variation  of  the  spe¬ 
cific  criterion  example  presented  here. 


The  discussion  above  provides  an  insight  into  the  roles  and  activities  an 
SCE  team  plays  in  RFP  generation.  However,  the  overall  impact  of  ex¬ 
plicit  instructions  depicted  must  be  assessed  by  the  development  orga¬ 
nization  expecting  to  undergo  an  SCE.  Organizations  that  have  not 
embarked  upon  a  software  process  improvement  program  or  have  never 
undergone  an  SCE  have  learned  that  RFP  release  is  not  the  time  to  be¬ 
gin.  It  will  generally  be  too  late  to  show  any  substantive  progress  and 
their  presentation  to  the  SCE  team  in  the  form  of  interview  responses 
and  documentation  may  be  less  than  desired. 

Of  note  is  a  recent  trend  toward  third-party  SCEs.  On  two  proposed  so¬ 
licitations,  a  government  agency  has  requested  that  bidding  develop¬ 
ment  organizations  submit  basic  information  regarding  their  experience 
of  having  undergone  an  SCE: 

•  SCE  conducted  by  trained  (SEI  or  SEI  Licensee)  team 

•  SCE  conducted  involving  programs  or  projects  which  will  be 
responsible  for  new  contract  work 

•  Name  of  evaluated  corporation,  division,  projects 

•  Dates  and  location(s)  of  onsite  portion  of  SCE 

•  Identification  of  organization  POC  and  SCE  team  leader  and 
organization  performing  SCE. 
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In  one  instance,  it  appears  that  the  agency  is  qualifying  organizations  by 
potentially  accepting  the  SCE  data  already  collected  or  by  performing  an 
additional  SCE.  In  another  instance,  it  appears  the  agency  is  attempting 
to  enable  itself  to  use  SCE  data  that  has  been  determined  in  the  recent 
past  (one  year)  for  current  source  selection  applicability. 
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3.5  Activity  5  Anaiyze  Instrument  Data 

Table  3-9  provides  an  overview  of  this  activity. 


1  Generic  SCE 

1  Supplier  Selection  SCE 

Purpose 

Identify  issue  areas  for  further  Investigation 
during  evaluation  conduct,  and  to  get  a 
preliminary  understanding  of  the 
organization’s  operations. 

Same  as  Generic  SCE  with  acquisition 
context. 

Actions 

Receive,  summarize,  and  analyze  instrument 
data.  Develops  profile(s)  analyses. 

Receipt  of  instrument  data  normally 
accompanies  proposal  receipt. 

Evaluation  of  proposals  initiated.  Competitive 
Range  Determination. 

Offerors  SCE  information  analyzed  for 
establishing  “general”  prioritization  of 
reference  model  components  for  all  offerors. 

Outcome 

The  team  has  a  high  level  understanding  of 
the  site’s  operations. 

The  team  has  a  high  level  understanding  of 
the  all  the  offeror’s  site  operations. 

Table  3-9:  Analyze  Instrument  Data  Overview 


3.5.1  Evaluator 

3.5.1. 1  Maturity  Questionnaire 

“Instrument,”  in  the  context  of  SCE,  is  a  data-collection  mechanism. 
What  distinguishes  instruments  from  other  data  collection  mechanisms 
are  the  formal,  written  nature  of  the  information  request  and  the  fact  the 
organization  has  some  time  to  research  and  prepare  the  response.  The 
primary  instrument  used  by  SCE  is  the  SEI  Maturity  Questionnaire.  The 
CMM-based  maturity  questionnaire  is  a  highly  structured  instrument  that 
was  formally  developed  by  the  SEI  and  is  rigidly  validated  and  re-validat¬ 
ed  through  empirical  methods  research.  This  aliows  a  level  of  compara¬ 
bility  and  reliability  beyond  that  of  other  instruments. 

3.5.1. 2  Site  Information  Packet 

The  other  instruments  SCE  uses  are  the  Product  Profiles  and  the  Site 
Information  packet  (SIP).  Nominally  the  instructions  contained  in  the  so¬ 
licitation  (RFP)  as  depicted  in  the  previous  section  dictate  the  content 
and  format  for  completion  and  return  of  these  instruments.  Specification 
in  the  instructions  for  the  Product  Profile  for  the  “proposed  work”  of  the 
acquisition  can  be  expected.  This  Product  Profile  will  be  compared  to  the 
sponsoring  organization’s  (e.g.,  government)  “profile  of  the  producf  to 
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be  acquired.  The  profile  template  attributes  are  identical  and  whether  a 
Product  Profile  is  “proposed”  or  of  a  current  existing  product  being  de¬ 
veloped  by  a  project  depends  upon  the  organization  creating  the  profile 
and  for  what  purpose  (e.g.,  source  selection,  contract  monitoring,  inter¬ 
nal  evaluation). 

The  primary  use  of  these  instruments  is  to  guide  the  data  collection  ef¬ 
forts  of  the  SCE  team.  The  SCE  method  calls  for  analysis  in  detail  of  the 
maturity  questionnaire  and  use  of  the  Product  Profiles  and  SIP  in  plan¬ 
ning  and  execution  of  the  overall  data  collection  effort. 

Summarization  of  maturity  questionnaire  responses  is  an  activity  of  no 
light  consequence.  Depending  upon  the  number  of  the  respondents; 
summarization  could  be  simple  manual  tabulation  or  require  automated 
support  to  provide  timely  information  for  use  by  the  team.  The  maturity 
questionnaire  is  designed  to  be  processed  both  manually  and  by  auto¬ 
mated  means. 

A  recommended  item  for  the  SIP  is  a  glossary  of  organizational  terminol¬ 
ogy  that  has  been  specifically  translated  with  the  CMM  in  mind.  The  SCE 
team  must  recognize  that  terminology  may  appear  identical,  but  have 
wholly  different  meanings  and  applications  in  different  organizations. 

A  typical  example  is  the  CMM  term  “peer  reviews.”  Although  this  is  Key 
Process  Area  unto  itself  and  encompasses  the  generalized  activities  of 
checking  one  person’s  or  group’s  work  in  a  formal  or  informal  manner, 
in  some  organizations  this  term  has  been  found  to  mean  a  highly  struc¬ 
tured  personnel  performance  review.  Alternatively,  many  organizations 
typically  carry  out  the  activities  described  in  the  CMM  Peer  Review  KPA 
as  “Inspections.” 

Note:  Following  the  formal  proposal  submission  due  date,  the  SSEB’s 
first  task  is  to  perform  an  initial  evaluation  and  determine  the  “the  pro¬ 
posal’s  responsiveness”  to  the  solicitation’s  requirements.  This  essen¬ 
tially  means  that  proposals  are  checked  for  compliance  with  the 
solicitation’s  (RFP’s)  instructions  and  an  initial  evaluation  of  whether  the 
proposals  are  viable  in  meeting  overall  requirements  and  warrant  further 
evaluation.  This  anaiysis  effectively  supplies  the  SSEB  the  information 
for  establishing  the  competitive  range.  Some  or  none  of  the  proposal  of¬ 
ferors  may  be  eliminated.  This  is  not  normally  communicated  to  the  off¬ 
erors.  Only  offeror’s  whose  proposals  are  deemed  “non-responsive,” 
and  judged  as  not  being  able  to  become  responsive  in  a  timely  manner, 
and  are  eliminated  would  receive  notification  of  this  event.  Establishing 
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the  competitive  range  ultimately  defines  the  number  of  site  visits  an  SCE 
team  will  need  to  accommodate  In  a  government  source  selection.  In  the 
commercial  environment  an  analogous  event  would  occur  with  much 
less  formality. 

3.5.2  Recipient 

It  is  incumbent  upon  the  recipient  organization  to  complete  the  “instru- 
menf  requested  information  as  accurately  as  is  feasible.  Recognize  that 
this  information  is  the  “First  Impression”  that  the  SCE  team  will  have  of 
the  organization  it  will  be  evaluating.  The  “picture”  conveyed  through  the 
SIP  and  the  MQ  should  be  organizationally  clear  and  coherent  as  is  pos¬ 
sible.  Close  adherence  to  the  instructions  and  terminology  definitions 
provided  with  the  MQ  will  provide  more  accurate  understanding  of  the 
correct  manner  to  respond  as  well  as  the  type  of  comments  to  add.  The 
better  the  SCE  team  is  able  to  grasp  and  understand  the  organizational 
structure  and  relationships  the  more  meaningful  the  MQ  data  will  be¬ 
come  and  the  more  accurately  their  data  collection  activities  will  focus 
on  those  issues  that  are  germane  to  the  solicitation. 
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3.6  Activity  6  Seiect  and  Prepare  Participants 

Table  3-10  provides  an  overview  of  this  activity. 


1  Generic  SCE 

1  Supplier  Selection  SCE  I 

Purpose 

Ensure  the  most  appropriate  personnel 
participate  in  the  evaluation,  and  ensure  that 
they  understand  the  evaluation  process  and 
shared  expectations. 

Ensure  that  the  most  representative 
personnel,  projects  and  sites  relative  to  the 
proposed  effort  for  the  acquisition  are 
investigated  and  the  evaluation  is 
appropriately  focused  for  each  offer’s  site. 

Actions 

Select  sites  to  conduct  evaluation,  select 
projects  to  Investigate,  select  participants  and 
prepare  and  conduct  initial  briefing. 

Evaluation  of  Proposals  continues. 

Offerors  projects  are  selected  to  undergo 

SCE.  Specific  preparation  for  individual 
offerors  initiated.  Logistical  coordination 
initiated. 

Outcome 

Site  participants  understand  the  evaluation 
process  and  are  ready  to  take  part. 

The  SCE  team  has  formulated  an  initial 
interview  strategy,  schedule  and  entry  briefing 
for  each  offeror  site. 

Table  3-10:  Select  and  Prepare  Participants  Overview 


3.6.1  Evaluator: 

Activity  6  (Select  and  Prepare  Participants)  and  Activity  7  (Prepare  for 
Data  Collection)  compose  the  Major  Activity  Grouping:  Specific  Prepa¬ 
ration.  These  activities  refine  the  preparation  activities  to  a  particular  de¬ 
velopment  organization’s  site.  The  purpose  of  these  two  activities  in  the 
Specific  Preparation  grouping  is  to  prepare  the  SCE  team  for  a  specific 
site  visit.  Activity  6  identifies  the  specific  organizational  sites(s), 
project(s)  and  participants  in  the  appraisal.  A  schedule  and  set  of  brief¬ 
ings  describing  the  appraisal  activities  will  be  prepared  and  delivered. 
Activity  7  (Prepare  for  Data  Collection)  prioritizes  topic  areas,  creates  an 
interview  strategy  in  consonance  with  the  overall  data  collection  effort 
and  finalizes  all  logistical  arrangements. 

For  acquisition,  one  of  the  essential  considerations  at  this  point  is  to  an¬ 
ticipate  teaming  arrangements  of  the  various  offerors.  These  arrange¬ 
ments  will  have  a  bearing  on  actual  projects  and  personnel  selected  to 
participate  in  the  SCE. 

3.6.1 .1  Teaming  Arrangements 

The  SCE  method  is  designed  to  handle  a  wide  range  of  organizational 
possibilities  that  offerors  propose.  Current  practice  on  most  software 
systems  range  from  the  simplest  cases  where  a  single,  localized  organi¬ 
zation  develops  all  the  software  to  the  most  complex  cases  where  mul- 
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tiple  contractors,  in  some  interrelationship,  participate  in  the 
development  and  integration  of  the  software  during  the  specified  life  of 
the  contractThis  section  illustrates  representative  cases  of  teaming, 
partnering,  and  subcontracting,  and  recommends  how  the  SCE  method 
might  be  applied  to  each  one.  It  should  be  noted  that,  for  a  given  pro¬ 
curement,  different  offerors  may  propose  different  organizational  ar¬ 
rangements.  Thus,  the  approaches  outlined  below  may  have  to  be 
tailored  and  combined  for  application  to  a  given  source  selection.  For  all 
situations,  several  key  ground  rules  should  be  observed: 

•  All  major  software  developers  should  be  evaluated. 

•  Questions  about  unique  or  different  processes  should  be 
answered  individually  by  the  participants. 

•  Even  when  an  offeror  proposes  common  processes,  evidence 
should  be  provided  by  individual  organizations. 

Single  Organization  as  a  Single  Team 

In  this  scenario,  the  software  is  developed  and  integrated  by  a  single 
contractor,  within  a  single  organization,  and  at  a  single  site.  The  SCE  is 
applied  to  this  single  organization. 

Multiple  Organizations  as  a  Single  Team 

In  this  scenario,  the  software  is  developed  and  integrated  by  multiple 
contractors,  within  multiple  organizations,  and  possibly  at  multiple  sites. 
The  various  parties  a/e  highly  merged  as  a  team,  and  the  contractors, 
organizations,  and  sites  are  all  known  at  the  time  of  the  source  selection. 
In  this  case,  the  SCE  is  applied  to  the  whole  team.  The  focus  must  be 
on  how  the  team,  as  a  cohesive  unit,  plans  to  do  business,  rather  than 
on  the  specific  individual  capabilities  of  the  various  team  participants. 
Typically,  a  single  set  of  data  is  collected  and  the  site  is  at  a  single  loca¬ 
tion  chosen  by  the  contractor  team.  If  the  particular  teaming  arrange¬ 
ment  is  new,  there  may  be  no  historical  data  on  how  well  the  combined 
team  capabilities  work.  Therefore,  evidence  must  be  collected  from  the 
various  team  participants,  and  the  evaluation  by  the  SCE  team  will  re¬ 
quire  considerable  engineering  judgement.  As  is  in  all  cases  where  de¬ 
tailed,  applicable  evidence  is  not  readily  available,  the  offerors  must  be 
able  to  demonstrate  why  their  selected  approach  was  chosen  among  the 
alternatives. 
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Single  Integrator  and  Developer  with  Suppliers  or  Vendors 

In  this  scenario,  the  major  items  of  software  are  developed  and  integrat¬ 
ed  by  a  single  contractor  or  team,  but  specific,  relatively  minor  or  local¬ 
ized  items  may  be  acquired  from  suppliers  or  vendors.  The  developing 
and  integrating  contractor  or  team  is  known  at  the  time  of  the  source  se¬ 
lection,  but  some  suppliers  and  typically  all  of  the  vendors  are  selected 
by  the  lead  team  at  some  later  time.  For  this  case,  the  SCE  should  be 
applied  to  the  contractor  or  team  as  it  is  known  at  the  time  of  the  source 
selection.  In  addition,  special  emphasis  must  be  placed  on  evaluating 
how  the  suppliers  or  vendors  will  be  selected  and  on  how  their  process¬ 
es  and  products  will  be  integrated  into  the  mainline  effort.  For  example, 
the  lead  contractor  team  might  use  the  technical  information  of  the  SCE 
method  (without  the  government  source  selection-specific  items)  to  con¬ 
duct  pre-selection  evaluation  of  its  suppliers.  Alternatively,  the  items  ac¬ 
quired  may  be  purchased  essentially  “off  the  shelf,”  and  detailed 
evaluation  of  the  vendor’s  development  capability  would  not  be  cost  ef¬ 
fective. 

Prime/Integrator  with  Multiple  Subcontractors 

In  this  scenario,  the  prime  contractoi  performs  the  integration  function, 
and  possibly  some  of  the  development,  but  major  portions  of  the  soft¬ 
ware  are  developed  by  subcontractors.  Some  of  the  subcontractors  may 
be  known  at  the  time  of  the  source  selection  and  some  may  be  sched¬ 
uled  for  later  selection.The  known  subcontractors  may  be  organized  to 
work  closely  with  the  prime  contractor  as  team  participants  or  may  plan 
to  work  somewhat  independently.  Some  aspects  of  this  case  are  analo¬ 
gous  to  the  other  cases  described  above  and  should  be  dealt  with  as 
outlined  in  those  descriptions.  However,  there  is  a  new  possibility  in  this 
arrangement  not  covered  by  the  previous  descriptions:  the  known  sub¬ 
contractors  may  not  be  highly  merged  into  the  lead  team.ln  this  case,  it 
is  recommended  that  separate  sets  of  data,  focused  on  the  technical 
and  managerial  content  of  their  assigned  portion  of  the  whole,  be  ob¬ 
tained  from  each  of  the  team  participants.  Separate  site  visits  are  also 
recommended  for  each  site  or  organization  involved.  Arrangements  for 
performing  site  visits  with  subcontractors  must  be  made  through  the 
prime  contractor.  The  prime  contractor  is  legally  entitled  to  be  involved 
and  must  be  invited  to  the  site  visit  and  allowed  to  participate  in  the  in¬ 
teraction  with  the  subcontractor.  However,  the  prime  contractor  repre¬ 
sentative  is  not  a  member  of  the  SCE  team  and  cannot  be  allowed  to 
participate  in  the  preparation  of  results  or  in  making  judgements  relative 
to  the  source  selection. 
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For  teaming  arrangements  in  which  subcontractors  will  be  developing 
relatively  minor  or  localized  portions  of  the  software,  it  may  be  appropri¬ 
ate  for  the  prime  contractor  to  invite  the  subcontractors  to  participate  in 
the  site  visit  to  the  prime,  in  lieu  of  site  visits  to  each  the  subcontractors. 
In  this  case,  the  prime  contractor  may  wish  to  organize  the  agenda  in 
such  a  way  that  the  subcontractors  can  participate  in  the  portions  that 
are  relevant  to  them  and  then  be  excused  from  proceedings  that  do  not 
involve  them  or  that  may  cover  items  that  are  considered  proprietary  to 
the  prime  contractor. 

3.6.1 .2  Entry  Briefing 

An  Initial  Briefing  should  be  prepared.  This  briefing  is  used  to  set  the  of¬ 
feror’s  expectations  for  the  site  visit.  At  a  minimum  the  briefing  should 

•  introduce  the  SCE  team  members 

•  delineate  the  on-site  schedule 

•  describe  whether  findings  or  results  will  be  debriefed 

•  set  any  operating  rules  for  interviewing  and  document  review 


3.6.2  Recipient 

The  various  alternatives  outlined  in  the  above  section  necessitate  the  of¬ 
feror  to  clearly  communicate  what  interrelationships  exist  and  how  the 
“offeror’s  team’s”  processes  will  facilitate  successful  execution  of  the 
contract  if  awarded.  Teaming,  subcontracting  relationships  that  are  in¬ 
complete  and  result  in  the  offeror  team  being  able  to  present  incomplete 
data  and  information  will  correspondingly  be  evaluated  as  having  incom¬ 
plete  processes  and  procedures  resulting  in  substantial  risk  to  the  agen¬ 
cy/organization  soliciting  for  products  and  services.  The  engineering 
adage  “Up  Front  and  Early”  is  most  appropriate  in  competing  for  con¬ 
tracts  involving  teaming,  subcontracting  relationships. 

At  a  lower  level  of  detail,  the  actual  participants  expected  to  be  inter¬ 
viewed  should  represent  the  most  experienced  and  knowledgeable  indi¬ 
viduals  for  their  respective  areas.  This  environment  can  become 
particularly  complicated  depending  upon  the  actual  situation  that  is  ex¬ 
pected  to  exist,  single  site,  teaming,  subcontractors  etc.  Accommodating 
the  appraisal  must  be  balanced  with  necessary  logistics  and  resources 
required  to  bring  appropriate  personnel  and  documentation  to  the  ap¬ 
praisal  site  versus  the  appraisal  team  visiting  multiple  offeror  develop¬ 
ment  sites. 
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Preparation  of  participants  should  include  at  a  minimum: 

•  identification  of  each  participant’s  role,  e.g.,: 

•  practitioner 

•  functional  area  representative 

•  technical  lead 

•  program/project  manager 

•  software  manager 

•  a  background  briefing  on  the  SCE  appraisal  method  and  its 
activities 

•  identification  of  documentation  appropriate  for  participants  to 
be  familiar  with 

Attendance  of  all  participants  at  the  Initial  Briefing  for  the  SCE  Team  at 
the  beginning  of  the  on-site  period  should  be  mandatory. 
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3.7  Activity  7  Prepare  for  Data  Coiiection 

Table  3-1 1  provides  an  overview  of  this  activity. 


1  Generic  SCE 

Supplier  Selection  SCE 

Purpose 

Plan  the  detailed  site  intervention  to  make 
optimum  use  of  available  site  visit  time  to 
attain  evaluation  goals  and  objectives. 

Plan  the  detailed  site  investigation  for  each 
offeror  site  to  make  optimum  use  of  available 
site  visit  time  to  attain  evaluation  goals  and 
objectives. 

Actions 

Prioritize  Focus  Areas,  refining  final  interview, 
document  review  strategies  and  appraisal 
team  roles  and  responsibilities. 

Evaluation  of  Proposals  continues. 

Prioritization  of  Reference  Model  components 
and  data  collection  strategy  for  onsite 
completed.  Logistical  coordination  finalized. 

Outcome 

The  team  has  finalized  all  plans  and  logistics 
and  is  ready  to  conduct  the  site  visit. 

Same  as  Generic  SCE. 

Table  3-1 1 :  Prepare  for  Data  Collection  Overview 


3.7.1  Evaluator 

This  is  the  second  of  two  activities  in  the  major  activity  grouping;  Specific 
Preparation. 

The  primary  action  for  this  activity  centers  on  prioritization  of  the  refer¬ 
ence  model  components  '^process  areas  (in  the  case  of  the  CMM,  the 
process  areas  are  the  KPAs)  and  results  in  the  identification  of  "^topics 
for  each  of  the  KPA  goals  of  the  Target  Process  Capability  identified  in 
Activity  1 . 

Prepare  for  Data  Collection  activity  refines  the  information  planned,  an¬ 
alyzed  and  delineated  into  discrete  schedules  for  onsite  interview  and 
document  review  i.e.,  develop  a  data  collection  plan.  The  data  received 
from  the  offerors  in  Activity  5  (General  Preparation)  and  Activity  6  (Spe¬ 
cific  Preparation)  is  focused  upon  the  individual  sites  to  be  evaluated. 

.  3.7.1 .1  Develop  Data  Collection  Plan 

With  this  activity,  discrete  interview  strategies  that  identify  exactly  who 
will  be  interviewed  in  what  sequence  and  what  is  expected  to  be  gained 
are  developed. 

Similarly,  discrete  document  review  strategies  are  also  developed.  This 
effort  normally  culminates  in  lists  of  documents  to  be  immediately  avail¬ 
able  for  their  arrival  onsite  (Initial  Document  Review)  and  a  listing  of  the 
documents  and/or  artifacts  to  be  solicited  during  the  interview  sched- 
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ules.  The  document  review  strategy  will  typically  request  organizational 
/project  polices,  standards,  procedures,  directives,  and  project  specific 
artifacts  (e.g.,  minutes  of  meetings,  test  case  logs,  trouble  reports). 

SCE  team  roles  (e.g.,  librarian,  timekeeper,  KPA  monitor/mini-teams) 
are  firmed  up  and  schedules  communicated  to  the  offeror’s  site  POC. 

Remember  that  the  SCE  activities  are  not  necessarily  sequential  in  ex¬ 
ecution.  Although  Activity  2,  Develop  Appraisal  Plan,  is  a  crucial  founda¬ 
tional  activity,  and  provides  the  overall  structure  and  schedule  for  the 
conduct  of  the  SCE,  information  from  Activities  3  -  6  will  be  used  to  con¬ 
sistently  refine  and  update  that  plan. 

Recognition  of  the  logistical  vagaries  (e.g.,  time  zones,  availability  of 
flights,  geographical  location,  offeror  site  working  hours,  security  con¬ 
cerns)  in  scheduling  multiple  SCEs  with  different  geographical  sites  and 
time  zones  must  accommodate  the  team’s  ability  and  availability  to  stay 
focused  on  the  task  at  hand. 

This  is  the  culmination  of  the  Plan  and  Prepare  Phase  of  the  SCE  and 
arms  the  SCE  team  with  all  available  information  to  proceed  to  the  next 
phase.  Conduct  Appraisal  or  the  onsite  execution  for  each  offeror  to  be 
evaluated. 

Normally,  government  PCO’s,  commercial  contracts  or  legal  offices,  will 
provide  specific  information  regarding  communication  with  offerors  be¬ 
ing  evaluated.  This  can  range  from  the  very  formal  official  letter  to  a  sim¬ 
ple  phone  call. 


3.7.2  Recipient 

3.7.2.1  Designate  SCE  Participants 

Once  the  onsite  agenda  is  finalized  and  communicated,  the  offeror  must 
assemble  the  appropriate  individuals  to  participate  in  the  SCE  during  the 
site  visit.  The  offeror’s  response  team  should  include  corporate  and 
project  management,  members  of  the  proposal  team  and  members  of 
the  projects  selected  for  evaluations,  particularly  the  systems  and  soft¬ 
ware  engineering  leaders.  This  may  include  both  functional  representa¬ 
tives  as  well  as  appropriate  project  personnel  and  may  include 
representatives  of  the  projects  identified  in  the  project  data.  By  the  time 
the  site  visit  is  scheduled,  the  offeror  will  have  submitted  the  SCE  re¬ 
sponse  data  to  the  proposal.  The  individuals  who  prepared  the  SCE  pro¬ 
posal  data  are  the  appropriate  people  to  prepare  the  data  required  to 
support  the  site  visit  discussion  topics,  as  well  as  any  discussion  on  Clar- 
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ification  Requests  (CRs)  and  Deficiency  Reports  (DRs)  identified  by  the 
SCE  team  and  communicated  subsequent  to  the  site  visit.  CRs  and  DRs 
would  not  normaiiy  be  issued  during  an  SCE  onsite,  but  foliowing  the  on¬ 
site  period  in  the  formal  “Discussions”  phase  of  the  acquisition. 

In  preparing  presentation  material  for  the  site  visit,  the  offeror  should  be 
aware  that  additional  consideration  will  not  be  given  for  elaborate  brief¬ 
ing  material.  The  focus  should  be  on  content.  Generally,  black  and  white 
transparencies-presented  on  one  projector-are  preferred.  Documents 
referenced  as  substantiation  for  MQ  responses  should  be  appropriately 
referenced  and  identified  as  to  content  and  MQ  reference. 

3.7.2.2  Arrange  Facilities  for  SCE  Teams 

The  offeror  needs  to  make  the  support  facility  arrangements  to  accom¬ 
modate  the  SCE  site  visit.  These  should  include  an  adequately  sized 
conference  room,  working  tables,  chairs,  telephone,  copying,  restrooms, 
and  refreshments  nearby.  Similarly,  secretarial  support  could  be  provid¬ 
ed  by  the  offeror  during  the  visit  to  help  with  telephone  messages, 
schedule  coordination,  and  document  access.  The  objective  of  thorough 
preparation  is  to  minimize  distractions  so  that  all  on-site  time  is  focused 
on  the  SCE  data  collection  necessary  for  successful  source  selection. 
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3.8  Activity  8  Receive  Presentations 

Table  3-12  provides  an  overview  of  this  activity. 

1  Generic  SCE 

Supplier  Selection  SCE  I 

Purpose 

Refine/update  the  evaluation  team’s 
understanding  of  the  organization’s  software 
process  operations. 

Same  as  Generic  SCE. 

Actions 

SCE  team  receives  briefing  from  organization. 

Evaluation  of  Proposals  continues. 

Same  as  generic  SCE.  SCE  onsite  for  each 
offeror  Is  executed. 

Outcome 

The  evaluation  team  has  a  refined/updated 
understanding  of  the  organization’s  process 
operations. 

Same  as  Generic  SCE. 

Table  3-12:  Receive  Presentations  Overview 


3.8.1  Evaluator 

This  activity  begins  the  Conduct  Appraisal  phase.  For  acquisition  appli¬ 
cations  in  particular,  the  site  visit  entry  presentation  starts  the  SCE 
team’s  data  collection  and  consolidation  efforts.  The  organization  will 
have  received  instructions  via  the  logistical  preparations  as  to  what  is 
expected.  Generally,  the  development  organization  will  present  over¬ 
views  of: 

•  the  orge/nization  and  relationships  to  programs,  projects, 
functional  staff 

•  organization  processes 

•  documentation 

•  process  improvement  plans 


3.8.2  Recipient 

Normally,  the  development  organization  will  receive  instructions  that  in¬ 
dicate  the  above  type  of  desired  presentations  and  what  kind  of  presen¬ 
tation  is  nof  desired  (e.g.,  marketing,  or  recitation  of  standard  processes 
discernible  from  documentation). 

Presentations  that  describe  how  the  organization’s  processes  are  exe¬ 
cuted  by  the  selected  projects  and/or  descriptions  of  how  the  organiza¬ 
tion’s  alternative  practices  demonstrate  the  interrelationships  of  its 
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various  processes  and  procedures  from  a  reference  model  (e.g.,  CMM) 
perspective  will  aid  the  SCE  team  in  their  understanding  and  data  col¬ 
lection  efforts. 
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3.9  Activity  9  Review  Documents 

Table  3-13  provides  an  overview  of  this  activity. 


1  Generic  SCE 

Supplier  Selection  SCE 

Purpose 

Understand  processes  actually  implemented 
in  the  organization. 

Same  as  Generic  SCE. 

Actions 

Determine  information  needed,  request 
documents,  artifacts;  review  relative  to 

Evaluation  of  Proposals  continues. 

reference  model  components,  and  take  notes. 

Same  as  Generic  SCE.  SCE  onsite  for  each 
offeror  is  executed. 

Outcome 

Understand  processes  actually  implemented 
in  the  organization. 

Same  as  Generic  SCE. 

Table  3-13:  Review  Documents  Overview 


3.9.1  Evaluator 

Note  that  the  sponsoring  agency  may  delineate  specific  documents  to 
be  available  for  SCE  team  review  (see  Figure  3-14  on  page  69).  Addi¬ 
tionally,  document  review  becomes  one  of  the  primary  indicators  of  “ob¬ 
jective  evidence”  that  allows  development  of  observations  and  validation 
of  these  obsen/ations  into  findings  of  strengths,  weaknesses,  and  im¬ 
provement  activities  by  the  SCE  team.  As  one  of  the  four  discrete  data 
collection  mechanisms  (instruments,  presentations,  interviews,  and  doc¬ 
ument  review),  documents  and  artifacts  (e.g.,  meeting  minutes,  trouble 
logs)  provide  the  clearest  path  of  corroboration  and  validation  of  obser¬ 
vations  leading  to  findings  while  on  site. 


3.9.2  Recipient 

Recognizing  the  SCE  team’s  need  for  objective  evidence  to  support  and 
demonstrate  process  capability  enhances  the  organization’s  effort  to 
place  its  best  foot  forward.  Any  tool  (electronic  or  paper)  that  can  aid  in 
demonstrating  process  maturity  should  be  made  available  through  the 
site  point  of  contact  to  the  SCE  team.  As  with  most  evaluations  or  ap¬ 
praisals,  time  onsite  is  limited  so  the  quicker  the  SCE  team  is  able  to  dis¬ 
cover,  grasp,  and  understand  the  organization’s  processes,  the  more 
the  team  and  organization  can  benefit  from  the  overall  on-site  period. 

Experienced  teams  have  requested  that  interviewees  bring  day-to-day 
operating  procedures,  engineer’s  notebooks,  schedules,  and  other  arti¬ 
facts  that  they  use  in  the  daily  performance  of  their  jobs  to  the  interview 
session.  The  interviewees  are  encouraged  to  use  the  material  as  appro- 
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priate  in  answering  questions  and  may  be  requested  by  the  SCE  team 
to  leave  the  items  for  iater  review.  The  items  would  be  reviewed  and  sent 
back  in  a  matter  of  hours  or  the  next  morning.  This  approach,  although 
seemingly  burdensome,  enhances  the  team’s  ability  to  understand  the 
day  to  day  processes  as  practiced  and  provides  readily  available  objec¬ 
tive  evidence  of  what  was  discussed  during  the  interview  and  is  ob¬ 
served  during  document  review  sessions. 
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3.10  Activity  10  Conduct  Interviews 

Table  3-14  provides  an  overview  of  this  activity. 


H  Generic  SCE 

1  Supplier  Selection  SCE  I 

Purpose 

Understand  site  personnel  perspective  on 
processes  implemented  In  the  organization. 

Same  as  Generic  SCE. 

Actions 

Determine  information  needed,  select, 
request  interviewees,  ask  questions,  take 

Evaluation  of  Proposals  continues. 

notes. 

Same  as  Generic  SCE.  SCE  onsite  for  each 
offeror  is  executed. 

Outcome 

Understand  site  personnel  perspective  on 
processes  implemented  in  the  organization. 

Same  as  Generic  SCE, 

Table  3-14:  Conduct  Interviews  Overview 


3.10.1  Evaluator 

Historically,  SCE  interviews  have  been  conducted  as  a  “many  on  one" 
event  (i.e.,  many  interviewers  (SCE  team)  on  one  interviewee  [a  single 
individual]).  This  satisfied  the  procurement  officiais’  desire  for  “unbiased” 
input.  However,  this  also  provided  a  significant  intimidation  factor  to  jun¬ 
ior  practitioners  as  weli  as  to  the  recipient  organizations’  desire  for  ap¬ 
propriate  representation  to  avoid  misunderstanding  and  “mistakes.”  This 
approach  is  no  longer  recommended.  Simply  stated,  the  value  of  single 
individual  interviews  as  standard  practice  has  been  diminished  signifi¬ 
cantly  with  advent  of  utilizing  multiple  data  collection  mechanisms  and 
the  development  of  corroboration  and  validation  rules  for  observations 
and  findings  ondite. 

This  does  not  mean  that  “many  on  one”  interviews  are  inappropriate  for 
all  occasions.  “Many  on  one”  interviews  have  been  retained  as  an  explic¬ 
it  option  for  the  SCE  team  and  sponsoring  organization.  “Many  on  one 
interviews”  may  be  the  most  efficient  technique  in  dealing  with  CEOs, 
Vice  Presidents,  Division  level  managers,  and  Program  Managers.  The 
type  of  information  desired  from  the  appropriate  level  of  personnel  pro¬ 
vides  guidance  for  “many  on  one”  or  “many  on  many”  interviews.  Pro¬ 
cess  implementation  at  the  practitioner  level  may  call  for  a  “many  on 
many”  functional  area  type  grouping  of  personnel  from  across  the  select¬ 
ed  projects.  This  breadth  and  depth  of  personnel  may  be  significantly 
more  efficient  than  individual  “many  on  one”  interviews  whereby  only  a 
handful  of  individuals  can  be  interacted  with  in  a  finite  amount  of  time. 
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3.10.2  Recipient 

Regardless  of  the  discussion  and  guidance  provided  in  the  above  dis¬ 
cussion,  note  that  some  sponsoring  organizations  may  still  insist  on 
“many-on-one”  style  interviews  for  all  participants.  The  recipient  organi¬ 
zation  can  and  should  make  Its  desires  and  concerns  known  prior  to  the 
actual  onsite  if  interview  conduct  guidance  is  not  provided  in  advance. 
The  onsite  period  will  not  be  the  time  to  make  a  substantive  issue  of  the 
interview  approach  employed  by  the  sponsor’s  SCE  team.  The  key  to 
the  maintenance  of  cordial,  professional  relationships  is  having  this  is¬ 
sue  clearly  understood  by  all  in  advance  of  the  onsite  period. 
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3.11  Activity  11  Consolidate  Data 

Table  3-15  provides  an  overview  of  this  activity. 

1  Generic  SCE 

1  Supplier  Selection  SCE  | 

Purpose 

Transform  the  data  collected  into  formal  team 
observations  of  process  strength  and 
weaknesses  relative  to  the  reference  model 
(e.g.,  the  CMM). 

Same  as  Generic  SCE. 

Actions 

Organize  and  combine  data,  determine  data 
coverage  and  sufficiency,  review/revise  data 
collection  plan. 

Evaluation  of  Proposals  continues. 

Same  as  Generic  SCE.  SCE  onsite  for  each 
offeror  Is  executed. 

Outcome 

The  team  has  an  agreed  to  baseline  of 
information  known,  information  needed,  and 
the  strategy  to  obtain  needed  information. 

Same  as  Generic  SCE. 

Table  3-15:  Consolidate  Data  Overview 


3.11.1  Evaluator 

With  the  Consolidate  Data  activity,  all  data  collection  efforts  are  re¬ 
viewed  and  “bounced”  against  the  original  data  collection  plan  (i.e.,  doc¬ 
ument  review  strategy,  interview  strategy,  presentation  data,  and 
instruments).  In  the  source  selection  environment  it  is  prudent  to  take  a 
more  conservative  approach  with  the  use  of  instruments  (e.g..  Maturity 
Questionnaire).  The  approach  advocated  is  to  use  instrument  data  as 
called  for  in  the  method,  but  not  basing  validation  or  corroboration  of  ob¬ 
servations  on  this  same  data.  This  is  because  of  the  nature  of  how  these 
instruments  are  completed.  In  a  supplier  selection  the  natural  thrust  of 
the  respondents  in  a  development  organization  requested  to  complete  a 
publicly  available  document  with  little  or  no  guidance  is  to  “put  their  best 
foot  forward.”  This  tendency  will  present  a  different  picture  than  one  in 
which  the  instrument  was  administered  by  internal  personnel  familiar 
with  the  organization  and  its  workings  with  the  ability  to  provide  guidance 
with  respect  to  the  goals  and  objectives  of  the  data  collection  effort. 

This  same  general  rationale  applies  to  other  types  of  data  not  directly 
controlled  by  the  sponsoring  agency  (e.g.,  internal  assessment  results, 
past  contract  performance).  The  caution  to  apply  in  using  any  of  this  type 
of  data  is  understanding  its  applicability,  timeliness  and  source. 
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3.11.2  Recipient 

The  above  discussion  applies  conversely  to  the  recipient  that  may  have 
unrealistic  expectations  regarding  the  type  of  data  that  is  solicited  and 
will  be  accepted  by  the  sponsoring  organization.  Much  anecdotal  infor¬ 
mation  is  available  illustrating  the  steadfast  earnestness  with  which  de¬ 
velopment  organizations  use  the  instruments  (MQs),  however,  the 
bottom  line  is  the  ability  of  the  sponsoring  organization’s  team  to  wit¬ 
ness,  understand  and  obtain  objective  evidence  that  provides  sufficient 
corroboration  and  validation  of  reference  model  components  (CMM)  of 
the  process  capability  claimed  by  these  instruments. 
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3.12  Activity  12  Deliver  Draft  Findings 

Table  3-16  provides  an  overview  of  this  activity. 


1  Generic  SCE 

1  Supplier  Selection  SCE  | 

Purpose 

Validate  preliminary  team  observations,  build 
credibility  in  the  evaluation,  and  generate  buy- 
in  to  the  eventual  results. 

Same  as  Generic  SCE  If  allowed. 

Actions 

Prepare  and  present  draft  findings.  Solicit 
feedback,  take  notes. 

Evaluation  of  Proposals  continues. 

Same  as  Generic  SCE  if  allowed. 

SCE  onsite  for  each  offeror  Is  executed.  A 
decision  to  conduct  this  activity  Is  made 
during  Activity  2  Develop  Appraisal  Plan. 
Typically,  this  activity  will  NOT  be  conducted 
during  a  Source  Selection  SCE. 

Outcome 

The  quality  of  the  evaluation  data  and  results 
is  Improved,  and  credibility  and  buy-in  to  the 
evaluation  process  and  its  results  Is 
generated. 

Same  as  Generic  SCE  if  allowed. 

Table  3-16:  Deliver  Draft  Findings  Overview 


3.12.1  Evaluator 

For  the  source  selection  application  of  SCE,  this  activity  may  be  preclud¬ 
ed  by  the  sponsoring  organization  due  to  contractual  and  legal  con¬ 
straints. 

However,  this  activity  may  be  allowed  in  the  general  contract  monitoring 
application.  Here  the  immediate  feedback  would  be  conducive  to  pro¬ 
cess  improvement  in  general  and  to  clear  communications  between  the 
organizations  in  particular.  Although  individual  specifics  of  contract  mon¬ 
itoring  applications  will  vary  (i.e.,  baseline  SCE,  award  fee  measure¬ 
ment,  value  engineering  support)  the  basic  premise  remains  the  same: 
a  consistent  dialogue  between  the  sponsoring  organization  and  its  con¬ 
tracted  development  organization  regarding  continuous  process  im¬ 
provement.  These  “teamwork”  style  approaches  are  enhanced  with  this 
type  of  activity  where  the  validation  of  onsite  real-time  process  evalua¬ 
tion  results  involve  the  majority  of  the  participants  of  the  evaluation. 
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3.12.2  Recipient 

Although  individual  sponsoring  agencies  may  not  allow  this  specific  type 
of  onsite  interaction,  the  recipient  should  aggressively  pursue  feedback 
regarding  their  onsite  period  performance. 
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3.13  Activity  13  Make  Rating  Judgement 

Table  3-17  provides  an  overview  of  this  activity. 


1  Generic  SCE 

Supplier  Selection  SCE  I 

Purpose 

Make  decisions,  based  on  validated 
observations,  about  the  organization’s 
process  capability,  using  the  reference  model 
components. 

Same  as  Generic  SCE. 

Actions 

Judge  satisfaction  of  reference  model 
components:  Activities  Performed  Key 
Practices,  individual  Process  Area  Goals, 
Activities  Performed  Common  Feature, 
Institutionalization  Common  Features, 

Process  Area  Goals  as  a  set,  Overall  Process 
Areas.  Determine  Maturity  Level  (if  required). 

Evaluation  of  Proposals  continues. 

Same  as  Generic  SCE.  SCE  onsite  for  each 
offeror  is  executed. 

Outcome 

A  formal  rating  decision  for  each  reference 
model  component  which  was  planned  to  be 
rated,  and  for  which  the  team  obtained 
sufficient  data  to  meet  method  rules  for 
conducting  the  rating. 

Same  as  Generic  SCE. 

Table  3>17:  Make  Rating  Judgement  Overview 


3.13.1  Evaluator 

The  rating  judgements  approach  is  determined  with  selection  of  the  rat¬ 
ing  baseline  carried  out  in  Activity  1  Analyze  Requirements  and  planned 
for  in  Activity  2  Develop  Appraisal  Plan.  The  basic  decision  rests  upon 
which  of  the  two  rating  options  available  is  selected. 

•  Option  1  -  Full  model  and  organizational  scope.  This  requires: 

•  full  coverage  rating  through  maturity  level 

•  Option  2  -  Reduced  scope  in  the  model  or  in  the  organization 
(standard  SCE).  This  requires  full  coverage  and  rating  of 
model  components  that  are: 

•  specified  during  requirements  analysis  and 
pianning  (Activity  1  and  2)  and 

•  meet  the  rules  of  rating,  such  that: 

•  performing  a  maturity  level  rating  will  not  be  feasible 

•  full  coverage  of  specified  items  is  required 
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3.13.2  Recipient 

The  recipient  organization  shouid  realize  that  the  selection  of  one  of  the 
three  ■■♦rating  baseline  options  is  the  primary  driver  behind  the  entire 
evaluation  data  collection  effort.  Accordingly,  understanding  the  spon¬ 
soring  organization’s  goals  and  objectives  and  being  abie  to  respond  ap- 
propriateiy  will  enhance  the  recipient’s  ability  to  provide  the  most 
accurate  and  compelling  picture  regarding  process  capabiiity. 
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3.14  Activity  14  Deiiver  Finai  Findings  Presentation 

Table  3-18  provides  an  overview  of  this  activity. 

1  Generic  SCE 

Supplier  Selection  SCE  1 

Purpose 

Provide  a  clear  and  actionable  summation  of 
the  evaluation  results  to  the  organization. 

Same  as  Generic  SCE  (If  allowed  onsite). 

Actions 

Prepare  and  present  Final  Findings.  Close  out 
site  activities. 

Evaluation  of  Proposals  continues. 

Same  as  Generic  SCE  (if  allowed  onsite) 

Acquisition  Sponsor  may  not  allow  this  activity 
to  be  executed  until  after  contract  award.  A 
simple  “exit  or  close  ouf  briefing/meeting  may 
be  done  onsite  Instead. 

Outcome 

The  sponsor  and  the  evaluated  organization 
understand  and  accept  the  team’s  findings. 

Same  as  Generic  SCE. 

Table  3-18:  Deliver  Final  Findings  Presentation  Overview 


3.14.1  Evaluator 

Although  delivery  of  the  final  findings  to  the  development  organization  is 
the  preferred  approach  to  facilitate  process  improvement,  contractual 
and  legal  constraints  may  preclude  full  execution  of  this  activity.  Instead 
the  final  meeting  at  the  conclusion  of  the  site  visit  may  be  a  “thank  you” 
exit  briefing.  At  a  minimum,  the  SCE  team  should  thank  their  hosts  and 
provide  some  indication  of  when  the  findings  results  (outcomes)  and  in¬ 
formation  about  the  individual  development  organization’s  performance 
would  be  available,  who  to  contact  and  how  to  proceed. 

As  discussed  in  Activity  12  (Deliver  Draft  Findings),  the  process  monitor¬ 
ing  application  is  most  applicable  for  the  delivery  of  final  findings  at  the 
conclusion  of  the  site  visit.  Award  Fee  determination  or  competitive  in¬ 
centives  among  multiple  suppliers  might  delay  delivery  of  final  findings 
to  some  future  date. 

Findings  should  always  be  delivered  at  the  earliest  possible  time  within 
these  constraints. 

3.14.2  Recipient 

During  an  exit  brief,  the  development  organization  should  find  out  how 
to  obtain  the  results  from  the  on-site  visit  they  have  just  hosted. 
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3.1 5  Activity  1 5  Produce  Reports  and  Support  Foliow-On  Activities 

Table  3-1 9  provides  an  overview  of  this  activity. 


Purpose 


Actions 


Generic  SCE 


Produce  a  formal  baseline  of  the  evaluation 
conduct  and  results  for  the  sponsor  and  other 
stakeholders,  and  ensure  the  evaluation 
results  are  used  appropriately  to  achieve 
stated  business  objectives. 

Produce  Reports: 

-Findings 
-Outcomes 
-Evaluation  Data 
-Method  Evaluation 

Distribute  Reports,  preserve  and/or  dispose  of 
records.  Support  follow-on  activities. 


Supplier  Selection  SCE 


Same  as  Generic  SCE. 


SCE  findings/outcomes  are  submitted  to 
SSEB.  SCE  Team  consults  with  SSEB  if 
requested.  SCE  team  may  act  as  advisor  to 
SSAC  and  SSA. 

Source  Selection  Evaluation  Board  (SSEB) 
compares  data  collected  against  Evaluation 
Standard-  assigns  technical  rating  and  risk 
identification. 


Source  Selection  Advisory  Council  compares 
and  ranks  offeror  proposals  submits  Risk 
Assessment  to  Source  Selection  Authority 
(SSA). 

Source  Selection  Authority  makes  award 
decision. 


Outcome  A  formal  baseline  of  the  evaluation  conduct 
and  results  is  established  and  reports  are 
delivered  to  stakeholders. 

The  evaluation  results  are  used  to  support 
business  objectives. 


A  formal  baseline  of  the  evaluation  conduct 
and  results  is  established  and  reports  are 
made  part  of  the  acquisition  files.  Delivery  of 
results  in  what  format  is  at  the  option  of  the 
sponsor  agency. 

Same  as  Generic  SCE.  The  evaluation  results 
are  used  to  support  business  objectives. 


Table  3-19:  Produce  Reports  and  Support  Follow-On  Activities 

Overview 


3.15.1  Evaluator 

3.15.1.1  Final  Findings  Report 

In  an  acquisition  SCE,  the  results  are  not  “confidential”  in  that  the  spon¬ 
sor  is  an  outside  organization  from  the  recipient.  But  the  results  are  only 
known  to  the  sponsor  and  the  recipient.  Competing  organizations  do  not 
see  the  results. 
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The  findings  (sometimes  called  “final”)  report  is  an  essential  item  for 
closing  out  an  SCE,  because  it  documents  ali  activities  and  results  from 
the  team’s  execution  of  the  method.  The  findings  report  should  contain 
the  following  information: 

•  Development  organization(s)  Information  •  the  Product 
Profiles,  organization  charts,  and  other  site  information,  and 
questionnaire  responses. 

•  Ali  worksheets  and  checklists. 

•  Objective  evidence  which  serves  as  a  basis  for  findings.  This 
section  should  be  a  formal  description  of  the  evidence 
supporting  the  team’s  findings  rather  than  the  actuai 
evidence.  The  team  will  not  be  allowed  to  take  the  evidence 
with  them. 

•  Findings,  including  a  separate  sheet(s)  for  each  key  process 
area.  The  findings  sheets  should  include  references  to  the 
objective  evidence  which  support  them. 

•  Ratings  (for  all  model  components  rated). 

An  outcomes  report  includes  recommendations  for  use  of  the  appraisal 
results,  in  accordance  with  the  planned  use  of  the  results  defined  in  Ac¬ 
tivities  1  and  2.  In  the  acquisition  applications  (source  selection  and  con¬ 
tract  monitoring)  this  report  may  contain  the  translation  of  SCE  findings 
and  ratings  into  evaluated  criteria  for  the  SSEB  or  the  sponsoring  orga¬ 
nization  to  determine  award  fee  and/or  progress  on  process  improve¬ 
ment  plans.  (See //appendix  B  for  examples.) 

3.15.1.2  Data  Disposition 

Data  disposition  is  executed  in  this  activity  in  consonance  with  the  deci¬ 
sions  made  during  Activity  1  (Analyze  Requirements)  and  Activity  2  (De¬ 
velop  Appraisal  Plan).  This  may  include: 

•  destroying  all  data  and  paper  with  the  exception  of  that  listed 
above  as  necessary  for  the  report,  or 

•  collecting  all  paper  and  electronic  artifacts  for  turnover  to  the 
PCO. 

Regardless  of  the  specificity  of  the  disposition  (e.g.,  most  government 
agencies  have  different  data  disposition  ruies)  of  the  development  orga- 
nization(s)  data,  it  is  incumbent  upon  the  SCE  team  to  provide  informa¬ 
tion  in  the  report  that  can  be  understood  and  interpreted  six  months  to  a 
year  after  the  SCE  site  visit.  Regardless  of  how  the  information  is  sub¬ 
sequently  used — revisiting  award  fee  determinations,  or  a  protest. 
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etc. — an  accurate,  understandable  report  will  save  countless  hours  of 
reconstruction  and  interpretation  by  individuals  not  part  of  the  original 
SCE  team. 

Data  disposition  includes  debriefing  not  only  the  winner  of  the  source  se¬ 
lection,  but  also  the  losers.  Specific  strategies  for  having  a  formal  debrief 
and  delivery  of  the  respective  development  organization’s  (winners  and 
losers)  should  be  developed  and  executed. 


3.15.2  Recipient 

The  recipient  organization,  if  the  sponsor  agrees  and  if  it  is  planned  for, 
may  always  choose  to  make  the  results  known  outside  the  organization. 
At  a  high  level,  this  might  be  done  for  marketing  and  public  relations  rea¬ 
sons.  (This  assumes  the  recipient  organization  is  satisfied  with  the  re¬ 
sults.)  On  another  level,  not  publicizing  the  information  may  be 
advantageous  due  to  simultaneous  bidding  activities  that  could  be  jeop¬ 
ardized  if  results  were  published. 

The  recipient  should  anticipate  being  provided  information  of  when, 
where,  and  in  what  form  (e.g.,  debriefing,  report)  the  organization  will  re¬ 
ceive  the  results  from  a  SCE  on-site  visit. 
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Ability  to  perform:  One  of  five  common  features  in  the  CMM  for 
Software.  The  ability  to  perform  reflects  the  preconditions  that  must 
exist  in  the  project  or  organization  to  implement  the  software 
process  competently.  Ability  to  Perform  typically  Involves  the 
features  of  resources,  organization  structures,  and  training. 

Accuracy:  An  observation  is  considered  to  be  accurate  if  the 
appraisal  team  agrees  that  it  is  based  on  what  is  heard  and  seen, 
is  worded  appropriately,  and  is  correctly  categorized  and  classified. 
[Masters  95] 

Activities  performed:  One  of  five  common  features  in  the  CMM 
for  Software.  Activities  performed  describe  the  roles  and 
procedures  necessary  to  implement  a  key  process  area.  Activities 
performed  typically  involves  the  features  of  establishing  plans  and 
procedures,  performing  the  work,  tracking  it,  and  taking  corrective 
action. 

Activity:  A  key  practice  of  the  activities  performed  common  feature 
in  the  CMM  for  Software. 

Acquisition:  The  cradle-to-grave  life  cycle  of  a  system  or  product, 
and  one  of  the  primary  applications  of  the  SCE  method.  When  used 
during  the  pre-contract  award  phase  of  an  acquisition,  may  be 
called  a  source  selection  SCE,  in  reference  to  the  U.S.  Department 
of  Defense  (DoD)  term  for  the  process  of  selecting  a  supplier  in  an 
acquisition.  When  used  during  the  contract  execution  phase,  may 
be  called  a  process  monitoring  SCE.  The  purpose  of  a  supplier 
selection  SCE  is  to  provide  input  to  the  sponsor  on  the  process 
capability  of  one  or  more  development  organizations.  The  outcome 
from  a  supplier  selection  SCE  is  the  selection  of  the  best  value 
supplier  for  performance  of  a  planned  contract.  SCE  results  are 
just  one  aspect  considered  in  the  sponsor’s  decision.  (See 
acquisition  agency  and  sponsoring  organization.) 

Acquisition  agency:  An  organization  responsible  for  developing, 
delivering,  and  supporting  a  system  or  product.  Not  normally  the 
producer  of  the  product.  For  purposes  of  this  document,  an 
acquisition  agency  is  the  appraisal  sponsoring  organization  when 
applying  the  SCE  method  for  the  purpose  of  selecting  a  supplier. 
(See  sponsoring  organization.) 
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Alternative  practice:  Practices  which  are  implemented  differently 
from  those  described  in  the  reference  model  that  may  accomplish 
the  goals  of  a  process  area. 

Anomaly:  A  contradictory  response  to  the  same  question  on  the 
questionnaire,  or  from  other  data  collection  mechanisms,  by  two 
(or  more)  projects.  May  indicate  an  issue  that  needs  to  be  probed 
further.  Related  to  inconsistency. 

Applicable  standards:  An  attribute  used  in  SCE.  This  attribute 
indicates  the  government  or  commercial  development  and  quality 
standards  that  are  imposed  on  the  project  or  organization,  such  as 
DOD-STD-2167A,  DoD-STD-2168,  MIL-STD-1521B,  or  MIL-STD- 
498,  or  ISO  9000-3. 

Application  of  the  SCE  method:  Synonym  for  use  of  the  SCE 
method. 

Application  domain:  An  attribute  used  in  SCE.  An  application 
domain  is  “a  bounded  set  of  related  systems  (i.e.,  systems  that 
address  a  particular  type  of  problem).  Development  and 
maintenance  in  an  application  domain  usually  require  special  skills 
and/or  resources.  Examples  include  payroll  and  personnel 
systems,  command  and  control  systems,  compilers,  and  expert 
systems.”  [Paulk  93b]  For  SCE,  this  is  an  attribute  used  within  the 
various  profiles.  The  application  domain  attribute  indicates  the  area 
of  subject  matter  expertise  needed  to  translate  system 
requirements  into  software  requirements,  and  indicates  significant 
differences  in  the  engineering  practices  which  transform  the 
software  requirements  into  accepted  code. 

Appraisal:  An  expert  or  official  valuation  of  something.  [AMD  85] 
In  the  context  of  model-based  process  appraisals,  an  appraisal  is 
an  examination,  by  a  trained  team,  of  an  organization's  current 
practices  from  a  process  management  perspective.  This  is  a 
dynamic  concept— the  act  of  appraising  (contrast  with  appraisal 
method). 

Appraisal  constraints:  Constraints  that  affect  appraisal  conduct 
such  as  budget  limitations,  schedule  limitations,  and  resource 
limitations  (people  and  facilities).  [Masters  95] 

Appraisal  goals:  The  desired  outcome  of  an  appraisal  process. 
[Masters  95] 
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Appraisal  method:  The  documented  process  for  conducting  an 
evaluation  or  assessment  of  something.  Specific  to  SCE,  the 
sequence  of  steps  performed  for  evaluating  the  process  capability 
of  an  organization  relative  to  a  reference  model.  Also,  a  set  of 
activities,  tools,  and  techniques  used  by  people  to  appraise  the 
process  capability  of  an  organization  at  a  given  point  in  time.  An 
appraisal  method  describes  a  process — “a  sequence  of  steps 
performed  for  a  given  purpose.”  [IEEE]  The  term  appraisal  method 
typically  refers  to  the  method  itself,  but  may  also  be  used  to 
connote  the  method  and  its  associated  documentation  and  training 
materials. 

Appraisal  outputs:  Any  lasting  artifact  produced  by  the  team  in 
executing  the  appraisal.  In  SCE,  the  primary  output  from  the  site 
visit  is  the  set  of  findings.  Often  synonymous  with  appraisal  results, 
although  in  the  SCE  context  appraisal  outputs  is  a  broader  term, 
because  results  only  relate  to  the  findings  and  ratings  generated. 

Appraisal  reports:  The  set  of  documented  artifacts  created  by  the 
appraisal  team  as  a  result  of  conducting  an  appraisal.  These 
reports  include:  findings  briefings  and  reports,  an  outcomes  report, 
an  appraisal  data  report,  and  a  method  evaluation  report. 
Collectively,  they  form  the  official  record,  or  baseline,  of  the 
appraisal  for  subsequent  use  by  the  sponsor  or  other  stakeholders 
in  the  data  and/or  process  executed.  All  reports  are  generated  after 
the  conclusion  of  the  site  visit  except  for  the  findings  briefing. 

Appraisal  'requirements:  Appraisal  goals  and  constraints. 
[Master  95] 

Appraisal  risk:  Risk  is  a  measure  of  uncertainty  of  attaining  a  goal, 
objective,  or  requirement  pertaining  to  technical  performance,  cost, 
and  schedule.  Risk  level  is  categorized  by  the  probability  of 
occurrence  and  the  consequences  of  occurrence.  This  includes  the 
adverse  consequences  of  process  variability.  [MIL-STD-499B]  For 
SCE,  appraisal  risk  has  two  components:  technical  risk  inherent  in 
the  method  as  defined  or  tailored,  and  process  risk  in  executing  the 
method.  Appraisal  risk  is  manifested  in  the  likelihood  (probability) 
of  errors  in  the  results  (i.e.,  that  the  findings  and  ratings  are 
incorrect).  (See  rating  baseline.) 
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Appraisal  scope:  The  boundaries  of  the  investigation,  in  terms  of 
the  breadth  within  the  organization  and  the  depth  within  the 
reference  model  used.  The  organizational  entities  and  CMM 
components  selected  for  investigation.  [Masters  95]  (See 
organizational  scope  and  reference  model  scope.) 

Appraised  entity:  The  organizational  units  to  which  appraisal 
outputs  apply.  An  appraised  entity  may  be  any  portion  of  an 
organization  including  an  entire  company,  a  selected  business  unit, 
a  specific  geographic  site,  units  supporting  a  particular  product 
line,  units  involved  in  a  particular  type  of  service,  an  individual 
project,  or  a  multi-company  team.  [Masters  95] 

Artifact:  an  object  produced  or  shaped  by  human  workmanship. 
[AHD  85]  For  model  based  process  appraisals,  artifacts  are  the 
products  resulting  from  enacting  a  process. 

Attributes:  characteristics  of  a  software  product  or  project.  The 
attributes  used  in  SCE  are  defined  throughout  this  glossary  and  are 
discussed  in  another  appendix  of  the  method  description. 

Audit:  An  independent  examination  of  a  work  product  or  set  of 
work  products  to  determine  compliance  with  specifications, 
standards,  contractual  agreements,  or  other  criteria.  [Paulk  93b] 

Candidate  findings:  Synonym  for  observations.  Candidate 
findings  are  observations  for  which  there  is  not  yet  enough 
objective  evidence  to  make  a  decision  (an  unvalidated 
observation).  (6ee  observations.) 

Caucus:  A  meeting  in  which  the  team  analyzes  information  they 
have  learned  while  on  site  during  appraisal  conduct,  including 
interviews,  document  review,  and  presentations,  to  transform  data 
into  observations  and  finally  into  findings.  SCE  teams  routinely 
participate  in  caucuses,  or  team  meetings,  during  an  SCE  site  visit. 
These  caucuses  are  designed  to  help  achieve  consensus  among 
the  team  members.  SCE  team  members  analyze,  share,  and 
consolidate  information  in  order  to  reach  conclusions  about  what 
was  seen  and  heard  as  a  result  of  their  data  collection  activities. 
(See  consolidation.) 

Capability  Maturity  Model®*^*  (CMM®*^):  “A  description  of  the 
stages  through  which  software  organizations  evolve  as  they  define, 
implement,  measure,  control,  and  improve  their  software 
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processes.”  [Paulk  93b]  For  SCE  this  is  a  model  which  is  used  to 
evaluate  a  development  organization’s  process  capability.  (See 
maturity  model.) 

Commitment  to  perform:  One  of  five  common  features  in  the 
CMM  for  Software.  Commitment  to  perform  reflects  the  actions  that 
the  organization  must  take  to  ensure  that  the  process  is 
established  and  will  endure.  Commitment  to  perform  typically 
involves  the  features  of  establishing  organizational  policies  and 
senior  management  sponsorship.  A  commitment  is  a  pact  that  is 
freely  assumed,  visible,  and  expected  to  be  kept  by  all  parties. 
[Paulk  93b] 

Common  feature:  “An  attribute  that  indicates  whether  the 
implementation  and  institutionalization  of  a  key  practice  is 
effective,  repeatable,  and  lasting.”  [Paulk  93b]  There  are  five 
common  features  defined  for  CMM  v1.1:  commitment  to  perform, 
ability  to  perform,  activities  performed,  measurement  and  analysis, 
and  verifying  implementation. 

Competitive  range:  Key  term  relating  to  the  acquisition  use  of  the 
SCE  method  in  government  source  selection.  By  law  (10U.S.C. 
2304  [g])  written  or  oral  discussions  in  negotiated  procurements 
must  be  conducted  with  all  responsible  offerors  who  submit 
proposals  within  a  competitive  range.  The  determination  as  to 
which  proposals  are  not  in  the  competitive  range,  and  the  exclusion 
of  offerors  either  before  or  as  a  result  of  written  or  oral  discussions, 
will  be  made  by  the  Contracting  Officer,  subject  to  the  approval  of 
the  sponsor.  The  sponsor  may  designate  the  evaluation  team 
chairperson  to  accomplish  this  function. 

The  competitive  range  must  be  determined  after  evaluation  of  all 
proposals  received,  on  the  basis  of  price  or  cost,  technical,  and 
other  salient  factors  including  proposal  deficiencies  and  their 
potential  for  correction.  The  competitive  range  must  include  all 
proposals  which  have  a  reasonable  chance  of  being  selected.  The 
objective  is  not  to  eliminate  proposals  from  the  competitive  range, 
but  to  facilitate  competition  by  conducting  written  and  oral 
discussions  with  all  offerors  who  have  a  reasonable  chance  of 
being  selected  for  an  award.  [USAF  84] 


Capability  Maturity  Model  and  CMM  are  service  marks  of  Carnegie  Mellon  Uni¬ 
versity. 
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Consistency:  The  degree  of  uniformity,  standardization,  and 
freedom  from  contradiction  among  documents  or  system 
components.  Consistency  of  an  appraisal  method  refers  to  the 
ability  of  different  appraisal  teams  using  the  same  method  to 
conduct  appraisals  of  the  same  scope  to  produce  non-conflicting 
results.  [Masters  95] 

Consolidation:  The  decision  making  activity  in  the  iterative 
information  gathering,  organizing,  and  analyzing  components  of 
the  SCE  process.  The  activities  conducted  by  the  appraisal  team 
to  transform  raw  data  collected  from  the  recipient  organization  into 
observations  and  findings.  Consolidation  activities  occur 
throughout  the  site  visit. 

Contract  monitoring:  A  specific  application  of  the  SCE  method. 
Euphemism  for  process  monitoring.  Part  of  the  process  monitoring 
“family”  of  evaluations.  (See  process  monitoring.) 

Corroboration:  In  SCE,  a  synonym  for  confirmation.  All  appraisal 
observations  must  be  confirmed  by  information  from  different 
sources  and  different  data  gathering  sessions  prior  to  use  as 
findings.  This  is  sometimes  referred  to  in  the  SCE  method  as  rules 
for  confirming  observations. 

Coverage:  The  extent  to  which  data  gathered  fully  addresses 
reference  model  components,  organizational  units,  and  life  cycle 
phases  within  the  scope  of  an  appraisal.  [Masters  95]  For  SCE,  the 
link  between  coverage  and  rating  is  important.  One  or  more 
validated  observations  that  the  team  agrees  fully  cover  the  area  of 
investigation  and  meet  method  rules  for  corroboration  (multiple 
sources,  multiple  sessions,  documentation,  etc.)  are  said  to  be 
sufficient  for  rating  the  reference  model  items.  (See  sufficiency  for 
rating,  validation,  and  corroboration.) 

Customer:  An  attribute  in  SCE.  This  attribute  indicates  who  the 
development  is  being  done  for. 

Data:  Information,  especially  information  organized  for  analysis  or 
used  as  the  basis  for  a  decision.  [AHD  85] 

Data  collection:  The  method  activities  related  to  obtaining 
information  from  the  appraised  entity  for  the  purpose  of  evaluating 
process  capability.  Four  data  sources  are  used  in  the  SCE  method: 
interviews,  document  review,  presentations,  and  instruments. 
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Development  organization:  An  organization  that  develops  and/or 
maintains  software  products.  The  development  organization  is  the 
recipient  of  an  SCE. 

Development  organization  community:  All  of  the  development 
organizations  that  are  evaluated  during  an  acquisition  use  of  the 
method.  In  an  acquisition  these  are  the  offerors  (or  all  of  the 
offerors  remaining  after  a  competitive  range  determination),  and 
possibly  their  subcontractors. 

Development  team  approach:  An  attribute  used  in  SCE.  It  is 
related  to  how  the  developer  organizes  itself  to  produce  the 
system:  the  degree  to  which  various  groups  interact  and  are 
brought  to  bear  on  the  effort. 

Directive:  An  order  or  instruction  describing  actions  that  must  be 
performed  and  authorizing  their  performance. 

Document:  Any  lasting  representation  of  information  available  to 
the  people  doing  development  and  management  work.  A 
document  can  be  viewed  as  an  external  memory  for  people. 
Documents  can  be  paper  or  electronic.  Any  process  artifact  can  be 
considered  a  “document”  in  an  SCE. 

Document  review:  One  of  four  primary  data  collection  sources 
used  in  SCE.  The  process  of  examining  documents  to  find 
evidence  of  the  processes  used  by  a  development  organization. 
Documents  can  define  and  standardize  processes,  can  indicate 
commitment  to  use  the  processes,  can  provide  an  audit  trail  of 
processes  that  were  used,  and  can  reflect  data  about  process 
performance.  Three  levels  of  documents  are  reviewed  during  an 
SCE:  organization-level,  project-level,  and  implementation-level. 

Environment:  An  attribute  used  in  SCE.  it  refers  to  the  hardware, 
software,  and  telecommunications  environment  used  to  develop 
the  system. 

Evidence:  Data  on  which  a  judgment  or  conclusion  can  be  based. 
[AHD  85] 

Effective  process:  A  process  that  can  be  characterized  as 
practiced,  documented,  enforced,  trained,  measured,  and  capable 
of  being  improved.  [Paulk  93b] 
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Evolution:  A  gradual  process  in  which  something  changes  into  a 
different  and  usually  more  complex  or  better  form.  [AHD  85] 

Evaluator:  Evaluate,  to  examine  and  judge  carefully.  [AHD  85}.  In 
the  context  of  SCE,  evaluator  is  referring  to  the  individual  on  a  team 
performing  an  evaluation  on  behalf  of  a  sponsor. 

Fact:  A  statement  whose  content  can  be  verified  as  true  through 
the  senses.  [Masters  95] 

Feature:  One  of  a  set  of  process  attributes  that  provide  a  view  of 
“whether  the  implementation  and  institutionalization  of  a  key 
practice  are  effective,  repeatable,  and  lasting.”  [Paulk  93bi  The 
features  used  in  SCE  come  directly  from  the  common  features  of 
CMM  v1.1.  They  add  a  level  of  detail  that  is  appropriate  for 
generating  topics  for  investigation.  Examples  of  features  are 
policies,  resources,  and  training.  Features  are  listed  within  each 
common  feature  defined  in  this  glossary.  (See  common  feature.) 

Fidelity:  Faithfulness  to  obligations,  duties,  or  observances.  [AHD 
85]  Fidelity  in  an  appraisal  means  adhering  strictly  to  the  reference 
model  used  to  appraise  processes.  CMM  fidelity  refers  to  the  use 
of  CMM  components,  and  CMM  components  alone,  as  the  basis 
for  rating  an  organization's  software  process  maturity.  [Masters  95] 
A  method  shows  good  fidelity  if  it  is  consistent,  repeatable, 
accurate,  and  precise.  Its  results  are  thus  comparable  across  and 
within  organizations,  and  errors  are  minimized.  Fidelity  is  closely 
related  to  reliability. 

Findings:  Findings  are  the  primary  output  from  executing  the  SCE 
method.  Final  findings  are  used  to  develop  the  findings  briefing  and 
final  report.  Findings  are  validated  observations.  Findings  consist 
of  strengths,  weaknesses,  or  improvement  activities  in  one  of  the 
reference  model  components  within  the  scope  of  the  appraisal. 
Findings  may  also  be  generated  in  non-reference  model  areas 
from  data  that  does  not  directly  correspond  to  the  reference  model 
used,  but  that  are  significant  to  the  success  of  the  organization’s 
operations.  (See  results.) 

An  observation  that  has  been  accepted  by  the  team  as  valid. 
Findings  include  strengths,  weaknesses,  evidence  of  alternative 
practices,  and  evidence  of  non-applicable  practices.  A  set  of 
findings  should  be  accurate,  corroborated,  and  consistent  within 
itself.  [Masters  95] 
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Goal:  A  summary  of  the  key  practices  of  a  key  process  area  that 
can  be  used  to  determine  whether  an  organization  or  project  has 
effectively  implemented  the  key  process  area.  [Masters  95] 

IDEAL  approach:  A  systems  approach  or  life  cycle  framework  for 
implementing  process  improvement  activities.  IDEAL  stands  for 
the  five  phases  of  the  approach:  Initiating,  Diagnosing, 
Establishing,  Acting,  and  Leveraging.  [Radice  93] 

Implementation-level  documents:  The  third  of  three  levels  of 
documents  reviewed  during  an  SCE.  These  are  documents  which 
provide  an  audit  trail  of  processes  that  were  used,  and  can  be  used 
by  the  development  organization  to  collect  data  about  process 
performance. 

Improvement  activity:  A  process  improvement  that  is  not  yet 
institutionalized — ^for  example,  a  pilot  program  that  implements  a 
new  configuration  management  process.  In  SCE,  it  indicates 
potential  mitigation  of  risk  due  to  implemented  process.  In  this 
sense,  an  improvement  activity  is  a  weakness  that  if 
institutionalized  would  be  considered  a  strength. 

Inconsistency:  An  apparently  contradictory  response  from  the 
same  project  to  two  (or  more)  questions  on  the  questionnaire,  or 
from  other  data  collection  mechanisms,  that  relate  to  the  same 
process  area.  May  indicate  an  issue  that  needs  to  be  probed 
further.  Related  to  anomaly. 

Inference:  A  conclusion  based  on  a  fact.  They  are  not  facts.  In 
SCE,  strong  inferences  may  be  used  as  a  basis  for  observations, 
in  addition  to  facts.  Strong  inferences  are  readily  verifiable  by 
further  data  collection. 

Institutionalization:  The  building  of  infrastructure  and  corporate 
culture  that  support  methods,  practices,  and  procedures  so  that 
they  are  the  ongoing  way  of  doing  business,  even  after  those  who 
originally  defined  them  are  gone.  [Masters  95] 

Institutionalization  common  feature:  One  of  four  common 
features  in  the  CMM  for  Software  that  are  related  to 
institutionalizing  methods,  practices,  and  procedures:  commitment 
to  perform,  ability  to  perform,  measurement  and  analysis,  and 
verifying  implementation.  [Paulk  93b] 
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instrument:  One  of  four  primary  data  collection  sources  used  in 
SCE.  An  instrument  is  typically  a  questionnaire,  survey,  profile,  or 
other  written  item  used  to  collect  data.  Instrument  data  is  typically 
collected  and  analyzed  prior  to  the  site  visit. 

Internal  evaluation:  One  SCE  application  type.  Various  internal 
evaluation  uses  are  tailored  applications  of  the  SCE  method. 
Typical  internal  evaluation  uses  include:  process  baselining, 
process  improvement  progress  measurement,  process  audits,  and 
domain,  product  line,  or  project  specific  appraisals.  Preparing  for 
an  external,  customer  led  evaluation  is  often  a  reason  that  an 
organization  conducts  an  internal  evaluation.  Related  to  acquisition 
and  process  monitoring  SCE  applications. 

Interviewing:  One  of  four  primary  data  collection  sources  used  in 
SCE.  The  process  of  questioning  personnel  from  the  development 
organization  to  find  evidence  of  the  processes  used  by  the 
development  organization.  Interviews  provide  insight  into  how 
processes  are  implemented  and  show  the  extent  to  which 
processes  have  been  internalized  by  members  of  the  development 
organization. 

Judgment:  The  exercise  of  making  sound  and  reasonable 
decisions  (verb).  [AMD  85]  In  SCE,  judgments  refer  to  individual 
and  team  decisions  in  the  data  transformation  process  from  notes 
to  obsen/ations,  observations  to  findings,  and  findings  to  ratings. 
(See  notes,  observations,  findings,  and  ratings.) 

Key  Practice:  The  infrastructures  and  activities  that  contribute 
most  to  the  effective  implementation  and  institutionalization  of  a 
key  process  area.  [Paulk  93b] 

Key  process  area  (KPA):  “A  cluster  of  related  activities  that,  when 
performed  collectively,  achieve  a  set  of  goals  considered  important 
for  establishing  process  capability.”  [Paulk  93b]  Each  KPA 
contributes  to  the  environment  in  which  development  organizations 
create  software  products.  Within  the  CMM,  the  KPAs  are  organized 
into  five  basic  levels  of  process  maturity  to  describe  the 
progression  from  an  ad  hoc  software  process  to  one  that  is  well 
defined  and  can  act  as  a  stable  foundation  for  continuous  process 
improvement. 
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Language(s):  An  attribute  for  SCE.  This  attribute  indicates  the 
programming  languages  in  which  the  code  is  to  be  written,  or  in 
which  it  has  been  written. 

Mapping:  The  relationship  between  actual  practices  in  the 
software  process  implementation  and  the  process  areas  within  the 
reference  model  used. 

Maturity  ievei:  “A  well-defined  evolutionary  plateau  toward 
achieving  a  mature  software  process.”  [Paulk  93b] 

Maturity  model:  A  model  of  organizational  activity  used  for 
evaluating  a  development  organization’s  process  capability.  The 
maturity  model  has  a  defined  structure,  and  is  available  to  the 
public.The  maturity  model  used  in  SCE  V3.0  is  the  Capability 
Maturity  Model  (CMM)  for  Software  V1 .1 .  [Paulk  93a] 

Measurement  and  analysis:  One  of  five  common  features  in  the 
CMM  for  Software.  This  common  feature  describes  the  need  to 
measure  the  process  and  analyze  the  measurements. 
Measurement  and  analysis  typically  includes  the  feature  of 
examples  of  the  measurements  that  could  be  taken  to  determine 
the  status  and  effectiveness  of  the  Activities  Performed. 

Method:  A  means  or  manner  of  procedure,  especially  a  regular 
and  systematic  way  of  accomplishing  something.  [AHD  85]  An 
appraisal  method  consists  of  appraisal  activities,  processes,  and 
rating  strategies  along  with  associated  data  structures,  definitions, 
and  usage  instructions.  (See  appraisal  method.) 

Method  tailoring:  Making,  altering,  or  adapting  to  a  particular  end. 
[AHD  85]  In  SCE,  tailoring  refers  to  selecting  options,  based  on  the 
appraisal  goals,  that  may  affect  appraisal  risk.  The  selection 
process,  led  by  the  team  leader  during  appraisal  planning,  of 
refining  or  extending  the  standard,  or  baseline,  method  to  best  fit 
the  needs  of  the  sponsor  and  the  appraisal  goals  defined  during 
requirements  analysis.  In  SCE  the  principal  tailoring  options 
include  varying  the  organizational  scope,  reference  model  scope, 
and  rating  baseline.  These  options  in  turn  drive  lower  level  tailoring 
options  for  team  size,  skills  and  experience,  and  time  on  site.  There 
are  also  numerous  low  level  implementation  options  relating  to 
forms,  templates,  and  instruments  (appraisal  method  artifacts) 
available  for  conducting  the  appraisal. 
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Notes:  The  transcription  of  raw  input  data  (from  instruments, 
presentations,  interviews,  and  documents)  by  an  individual  team 
member,  usually  in  the  form  of  written  text,  into  information 
formatted  such  that  it  can  later  be  used  to  form  observations  about 
processes.  In  SCE,  the  formatting  is  done  by  various  means, 
including  “tagging”  notes  relative  to  the  reference  model  used. 

Observation:  An  inference  or  judgment  that  is  acquired  from  or 
based  on  observing.  [AMD  85]  An  observation  is  information 
extracted  from  the  notes  of  data  collection  sessions.  [Masters  95] 
Observations  are  classified  in  terms  of  strengths  and  weaknesses, 
and  categorized  by  reference  model  component.  In  SCE, 
observations  are  always  based  on  facts  or  strong  inferences. 

Organization-level  documents:  The  first  (or  top)  level  of  three 
levels  of  documents  reviewed  during  an  SCE.  These  are  the 
policies  and  procedures  which  establish  the  development 
environment  for  all  company  project  activities.  Organizational  level 
documents  define  the  process  and  management  constraints  the 
organization  places  on  projects. 

Organizational  scope:  The  part  of  the  appraisal  scope  that 
defines  the  breadth  of  the  investigation  within  the  development 
organization.  Typically  described  in  terms  of  a  project  or  number  of 
projects,  but  may  also  relate  to  a  product  line  or  domain.  The 
organizational  units  that  comprise  the  entity  being  appraised. 
[Masters  95]  (See  appraisal  scope.) 

Outcome:  How  the  findings  (SCE  results)  are  used  by  the 
sponsoring  organization — ^for  example,  in  risk  determination  for  an 
acquisition,  risk  management  for  process  monitoring,  or  process 
improvement  for  an  internal  evaluation. 

Policy:  “A  guiding  principle,  typically  established  by  senior 
management,  adopted  by  an  organization  to  influence  and 
determine  decisions.”  [Paulk  93b] 

Precedence:  An  attribute  used  in  SCE.  This  attribute  indicates 
whether  the  principal  stakeholders  in  the  system  (acquirer,  end 
user,  developer)  have  experience  with  the  type  of  system  to  be 
built.  Systems  that  are  providing  a  new  capability  tend  to  have 
more  changes  to  the  requirements  than  do  ones  that  are  replacing 
existing  systems. 
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Presentations:  One  of  four  primary  data  collection  sources  used 
in  SCE.  Presentations  can  either  be  delivered  by  the  appraisal 
team  to  the  recipient  organization,  or  can  be  delivered  by  the 
recipient  organization  to  the  appraisal  team.  Usually  these 
presentations  are  provided  in  a  viewgraph,  briefing  format  allowing 
interaction  between  the  team  and  the  participants.  Presentations 
can  delivered  either  for  the  purpose  of  data  collection  or  data 
validation.  (See  data  collection  and  validation.) 

Procedure:  A  written  description  of  a  course  of  action  to  be  taken 
to  perform  a  given  task.  [IEEE  91] 

Process:  A  sequence  of  steps  performed  for  a  given  purpose. 
[IEEE  91] 

Process  capability:  ‘The  range  of  expected  results  that  can  be 
achieved  by  following  a  process.”  [Paulk  93b] 

Process  maturity:  The  extent  to  which  a  specific  process  is 
explicitly  defined,  managed,  measured,  controlled,  and  effective. 
Maturity  implies  a  potential  for  growth  in  capability  and  indicates 
both  the  richness  of  an  organization's  software  process  and 
consistency  with  which  it  is  applied  in  projects  throughout  the 
organization.  [Paulk  93a] 

Process  monitoring:  One  of  the  primary  applications  of  the  SCE 
method.  In  ^irocess  monitoring,  SCE  results  can  serve  as  an  input 
for  an  incdntive/award  fee,  as  a  basis  for  value  engineering 
incentive  payments,  or  can  be  used  to  help  the  sponsoring 
organization  tailor  its  contract  monitoring  efforts. 

Procuring  Contracting  Officer  (PCO):  The  PCO  is  the  acquisition 
agency  person  responsible  for  all  communications  with  the  offerors 
(development  organizations)  in  an  acquisition  application  of  SCE. 
The  PCO  ensures  that  the  entire  source  selection  process  is 
consistent  with  applicable  regulations.  The  PCO  is  also 
responsible  for  advising  the  sponsor  on  the  interpretation  of  the 
findings  to  ensure  a  consistent  and  objective  award  decision. 

Product  profile:  See  Profiles. 

Product  type:  An  attribute  in  SCE.  The  product  type  attribute 
refers  to  the  particular  aspect  of  the  application  domain  which  the 
system  will  support  or  to  the  type  of  service  which  the  system  will 
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provide.  For  example,  displays  or  communications  could  be 
product  types  in  a  command  and  control  system,  a  weapons 
system,  or  another  application  domain.  Although  there  may  be 
similarities  in  the  communications  subsystem  in  the  various 
application  domains,  they  each  have  their  own  set  of  unique 
problems  which  must  be  addressed. 

Profiles:  A  profile  is  the  set  of  attributes  (such  as  Application 
Domain,  Product  Type,  and  Size)  associated  with  a  product  and 
the  environment  that  supports  development  of  the  product.  There 
are  three  types  of  product  profiles  used  in  SCE:  a  “targef  Product 
Profile  created  by  the  sponsor  organization,  representing  the 
customer  view  and  reflecting  a  “desired”  state;  Product  Profile(s) 
from  the  recipient  reflecting  attributes  of  a  current  effort(s);  and  a 
“proposed”  Product  Profile  created  by  the  offeror  in  response  to  an 
acquisition  application  reflecting  the  developer  view  of  planned 
work. 

Project:  An  undertaking  requiring  concerted  effort,  which  is 
focused  on  developing  and/or  maintaining  a  specific  product.  The 
product  may  include  hardware,  software  and  other  components. 
Typically  a  project  has  its  own  funding,  cost  accounting,  and 
delivery  schedule.  [Masters  95] 

Project-level  documents:  The  second  of  three  levels  of 
documents  reviewed  during  an  SCE.  These  are  documents  which 
define  the  development  processes  in  use  for  a  particular  project. 
Project  level  documents  define  the  detailed  processes  that  are 
used  to  manage,  coordinate,  and  integrate  the  engineering 
activities  required  for  the  development. 

Rating:  A  position  assigned  on  a  scale;  standing.  [AHD  85] 
Ratings  are  judgments  associated  with  findings.  A  characterization 
of  an  organization's  process  relative  to  a  component  of  the 
reference  model  used  in  the  appraisal.  Rating  types  in  SCE  include 
satisfied,  not  satisfied,  not  rated,  or  not  applicable.  The  rating  scale 
for  maturity  level  is  taken  directly  from  the  definition  contained  in 
the  reference  model  (e.g..  Levels  1-5  in  the  CMM  for  Software). 
Ratings  can  be  applied  to  any  component  of  the  reference  model 
that  is  planned  for  by  the  team  to  achieve  appraisal  goals  and  if 
collected  data  meets  all  method  rules  for  coverage  and 
corroboration.  (See  appraisal  goals,  coverage  and  corroboration.) 
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Rating  baseline:  “Base”  is  the  supporting  part  or  layer;  foundation. 
The  fundamental  principle  or  underlying  concept  of  a  system  or 
theory:  basis.  The  fact,  observation,  or  premise  from  which  a 
reasoning  process  is  begun.”  [AHD  85]  A  baseline  is  a  specification 
or  product  that  has  been  formally  reviewed  and  agreed  upon,  that 
thereafter  serves  as  the  basis  for  further  development...[Paulk  93b] 
In  SCE,  choosing  the  rating  baseline  option  is  the  fundamental 
method  tailoring  decision,  made  during  appraisal  requirements 
analysis.  This  decision  drives  subsequent  planning  and  execution 
of  the  method.  It  specifies  the  choice  of  method  “rigor”  made  by  the 
sponsor  (in  consultation  with  the  team  leader  or  senior  site 
manager).  It  reflects  the  reference  model  scope  and  coverage 
requirements  enabling  team  rating  judgments  to  be  made.  The 
SCE  method  provides  two  rating  baseline  options:  depth  oriented 
and  breadth  oriented  (See  Method  Description  for  more  detail.) 

Recipient:  The  appraised  entity  that  receives  the  appraisal. 
Synonymous  with  development  organization.  (See  appraised 
entity  and  development  organization.) 

Reference  model  scope:  The  part  of  the  appraisal  scope  that 
defines  the  depth  within  the  reference  model  used  that  will  be 
investigated  during  the  SCE.  Items  outside  the  defined  scope  of 
the  SCE  cannot  be  looked  at  during  an  acquisition  application  of 
SCE.  (See  appraisal  scope.) 

Reliability:  The  ability  of  a  system  or  component  to  perform  its 
required  functions  under  stated  conditions  for  a  specified  period  of 
tim.  [IEEE  90]  In  SCE,  the  method  is  the  “system.”  Reliability  is 
generally  used  to  refer  to  the  repeatability  and  consistency  of  the 
appraisal  method.  The  ability  to  attain  appraisal  results  that 
accurately  characterize  an  organization's  software  process. 
[Masters  95] 

Repeatability:  The  ability  to  attain  the  same  appraisal  results  if  an 
appraisal  of  identical  scope  is  conducted  more  than  once  in  the 
same  time  period.  [Masters  95] 

Request  for  Proposal  (RFP):  A  government  acquisition  document 
that  describes  characteristics  of  the  system  the  sponsor  wants  to 
acquire.  It  is  used  in  an  acquisition  application  of  the  SCE  method. 
This  document  is  used  to  solicit  proposals  from  commercial 
development  organizations  (offerors)  and  to  communicate  the 
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characteristics  of  the  desired  system  to  the  offerors.  In  source 
seiection,  this  is  the  document  that  specifies  that  an  SCE  wiil  be 
performed,  how  it  will  be  performed,  and  what  is  expected  of  the 
offerors  to  respond  to  the  customer’s  need.  The  RFP  is  a  key 
artifact  describing  the  appraisal  plan  in  an  acquisition  application. 

Results:  A  synonym  for  the  SCE  findings.  This  is  the  primary 
output  of  the  SCE.  In  addition  to  findings,  the  results  may  include 
reference  model  ratings  (such  as  maturity  level),  and  draft 
recommendations,  depending  on  the  application  of  the  method  and 
the  goals  documented  in  the  appraisal  plan.  Appraisal  results  are 
always  provided  to  the  sponsor,  and  should  be  provided  to  the 
appraisal  recipient  (development  organization).  (See  findings, 
appraisal  outputs.) 

Reuse  estimate:  An  attribute  used  in  SCE.  It  indicates  the 
development  organization’s  approach  to  building  the  product.  It  is 
correlated  to  the  size  attribute. 

Sampling:  A  set  of  elements  drawn  from  and  analyzed  to  estimate 
the  characteristics  of  a  population.  During  an  appraisal,  data 
collection  is  planned  to  provide  a  sampling  of  the  process  data 
related  to  the  reference  model  components,  organizational  units, 
and  life  cycle  phases  within  the  scope  of  the  appraisal.  [Masters  95] 

Site:  A  geographic  location  of  one  or  more  of  an  organization's 
units  that  participate  in  an  appraisal. 

Site  information  packet:  The  set  of  rhaterials  requested  by  the 
sponsor,  and  provided  to  the  appraisal  team  by  the  recipient 
organization,  for  use  in  planning  and  preparing  for  the  appraisal.  In 
SCE,  it  includes  information  such  as  organization  charts,  site 
terminology  lists,  document  hierarchy  and  model  content  mapping, 
product  profiles  (including  the  proposed  product  profile  for  an 
acquisition  SCE),  responses  to  instruments,  etc. 

Site  technical  coordinator:  The  technical  focal  point  assigned  by 
the  recipient  organization  to  assist  and  facilitate  the  appraisal  team 
in  conducting  its  activities. 

Site  visit:The  collection  of  SCE  activities  that  encompass  the 
investigation  by  the  SCE  team  at  a  development  organization’s 
site. 
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Size:  An  attribute  for  SCE.  The  size  attribute  indicates  the 
magnitude  of  the  product  (and  hence  the  required  project).  Size  is 
composed  of  three  related  attributes.  The  contract  duration  is  the 
estimated  or  required  length  of  time  for  the  development  of  the 
product.  The  team  size  is  the  number  of  developers  who  will  be 
involved  in  the  project.  The  estimated  size  is  the  amount  of  code  to 
be  developed  (in  a  software  system). 

Software  Capability  Evaluation  (SCE):  A  method  for  evaluating 
the  software  process  of  an  organization  to  gain  insight  into  its 
software  development  capability.  SCE  can  also  be  defined  as  a 
method  for  evaluating  the  processes  of  an  organization  to  gain 
insight  into  its  business  capability.  Which  model  processes  are 
evaluated  is  determined  by  the  sponsor  during  appraisal  planning 
(e.g.,  software,  people,  acquisition). 

Software  development  plan  (SDP):  ‘The  collection  of  plans  that 
describe  the  activities  to  be  performed  for  the  software  project.” 
[Paulk  93b] 

Software  process  capability:  “The  range  of  expected  results  that 
can  be  achieved  by  following  a  process.”  [Paulk  93b]  For  purposes 
of  an  SCE,  software  process  capability  reflects  those  processes 
which  provide  an  environment  for  development  teams  to  produce 
software  products.  The  processes  evaluated  include  decision 
making  processes  (such  as  software  project  planning), 
communication  processes  (such  as  intergroup  coordination)  and 
technical  processes  (such  as  peer  reviews).  (See  process 
capability  and  process  maturity.) 

Software  process  implementation:  A  tailored  set  of  practices 
that  defines  how  software  development  work  is  supposed  to  be 
done. 

Source  selection:  The  government  term  for  a  acquisition  process 
to  select  a  supplier.  An  acquisition  application  of  the  SCE  method 
is  used  to  provide  results  that  are  factored  into  the  source  selection 
decision.  In  source  selection,  the  results  of  the  SCE  are  used  by 
the  sponsoring  organization  to  characterize  the  process-related 
risk  of  awarding  a  contract  to  an  offeror.  SCE  is  only  one  criterion 
among  many  used  to  select  contractors  in  government 
acquisitions.  (See  acquisition  and  acquisition  agency.) 
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Source  Selection  Authority  (SSA):  The  individual  responsible  for 
the  conduct  of  the  government  source  selection  (acquisition) 
process.  In  an  acquisition  application  of  the  SCE  method,  the  SSA 
is  the  sponsor.  The  SSA  is  the  final  arbiter  on  the  use  of  SCE, 
approves  how  the  SCE  results  will  influence  the  award  decision, 
and  makes  the  award  decision.  (See  acquisition,  acquisition 
agency,  source  selection  advisory  council,  and  source  selection 
advisory  board.) 

Source  Selection  Advisory  Council  (SSAC):  The  SSAC  is 
chartered  by  the  sponsoring  organization  (acquisition  agency)  with 
collecting  and  analyzing  the  evaluations  of  each  offeror.  This  group 
performs  risk  assessment  activities.  This  is  the  only  group 
permitted  to  compare  the  SCE  results  (strengths  and  weaknesses) 
of  the  offerors  against  one  another.  The  SSAC  may  recommend  to 
the  sponsor  how  the  SCE  findings  will  be  incorporated  into  the 
award  decision  at  the  pre-RFP  release  briefing. 

Source  Selection  Evaluation  Board  (SSEB):  This  is  the 
government  group  that  evaluates  the  offerors’  proposals  against 
defined  evaluation  standards  in  an  acquisition  application  of  SCE. 
This  group  performs  risk  identification  tasks.  This  group  develops 
the  evaluation  standards  and  receives  approval  to  use  them  from 
sponsor  before  the  issuance  of  the  RFP.  The  SSEB  is  usually 
organized  into  technical  and  cost  teams  important  to  the  award 
decision.  If  the  ifindings  of  an  SCE  are  being  factored  into  the 
source  selection  decision  as  an  Evaluation  Criterion,  the  SCE  team 
leader  should  be  a  member  of  the  SSEB.  The  SSEB  prepares,  prior 
to  the  release  of  the  RFP,  an  evaluation  standard  that  will 
incorporate  the  SCE  results  into  the  source  selection  process. 

SSEB  Chairperson:  The  SSEB  chairperson  coordinates  all 
activities  of  the  SSEB  related  to  the  acquisition.  The  chairperson 
will  facilitate  the  incorporation  of  SCE  into  the  source  selection 
documentation  and  monitor  the  various  evaluation  teams,  including 
the  SCE  team. 

Sponsor:  The  decision  maker  in  the  organization  that 
commissions  the  SCE  to  be  performed  and  uses  the  findings 
(results).  Evaluator  results  are  always  provided  to  the  sponsor.  The 
individual  who  authorizes  an  evaluation,  defines  its  goals  and 
constraints,  and  commits  to  use  of  evaluation  outputs.  [Masters  95] 
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Sponsoring  organization:  The  organization  that  commissions  the 
SCE  to  be  performed  and  uses  the  findings.  (See  sponsor  and 
acquisition  agency.) 

Standard:  “Mandatory  requirements  employed  and  enforced  to 
prescribe  a  disciplined,  uniform  approach  to  software 
development.”  [Paulk  93b] 

Strength:  Indicates  the  team  judgment  of  an  effective 
implementation  of  a  component  of  the  reference  model.  In  SCE,  a 
strength  further  indicates  a  particular  part  of  the  organization’s 
capability  that  is  sufficiently  robust  to  mitigate  the  development 
risks  due  to  process. 

Implementation  of  practices  which  in  an  appraisal  team's 
judgment.  Improve  an  organization's  software  process  capability. 
CMM  related  strengths  are  effective  implementation  of  one  or  more 
of  the  CMM  key  practices  or  one  or  more  alternative  practices  that 
contribute  equivalently  to  the  satisfaction  of  KPA  goals.  [Masters 
95] 

Subcontractor:  A  development  organization  that  is  contracted  to 
work  for  another  development  organization  to  produce  products. 

Subcontractors:  An  attribute  in  SCE.  This  attribute  is  used  to 
indicate  whether  the  development  organization  intends  to  use 
subcontractors  in  the  development,  and  is  a  factor  if  they  lack 
experience  with  subcontract  management. 

Sufficiency  for  rating:  The  extent  to  which  observations  meet 
appraisal  method's  rules,  thus  satisfying  the  prerequisites  for 
rating.  Sufficiency  judgments  are  composed  of  a  series  of  team 
judgments  regarding  the  validation,  coverage,  and  corroboration 
aspects  of  observations. 

Target:  An  attribute  in  SCE.  This  attribute  indicates  the  hardware 
configuration  that  the  developed  software  will  run  on  when 
operational. 

Target  Process  Capability:  The  process  capability  that  is  most 
appropriate  for  the  planned  development;  the  process  capability 
desired  by  the  sponsoring  organization  for  the  product  to  be 
developed.  The  Target  Process  capability  consists  of  a  set  of 
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process  areas  within  the  reference  model  used,  and  heips 
establish  the  boundaries  of  the  SCE  investigation — ^a  process  area 
is  evaluated  if  and  only  if  it  is  part  of  the  Target  Process  Capabiiity. 

Topic:  A  topic  is  a  focused  subject  matter  area  probed  during  the 
SCE  investigation.  Topics  are  a  subset  of  process  activities  that 
work  towards  achieving  a  specific  process  area  goal.  Topics  are 
intended  to  be  detailed  enough  to  focus  the  investigation  on 
observable,  documented  work  practices,  but  sufficiently  abstract 
that  they  avoid  prescribing  how  the  process  area  is  implemented. 
Topics  are  selected  by  considering  the  intersection  of  a  process 
area  goal  and  its  associated  reference  model  features. 

Type  of  Work:  An  attribute  for  SCE.  This  attribute  indicates  the 
portion  of  the  development  life  cycle  which  will  be  performed.  As 
examples  of  different  types  of  work,  in  “full  software  developmenf 
a  development  organization  is  required  to  build  a  product  based  on 
the  system  requirements,  while  in  “code  development  only”  the 
development  organization  is  required  to  develop  code  according  to 
the  system  requirements  and  software  top  level  design  provided  by 
the  issuing  authority. 

Use  of  the  SCE  method:  Executing  the  SCE  method  within  a 
particuiar  context.  The  principal  high-level  uses  of  the  SCE  method 
are  in  acquisition,  and  process  monitoring,  and  internal  evaluation. 
This  is  sometimes  referred  to  as  the  application  of  the  method. 

Validation:  To  substantiate;  verify.  Valid  refers  to  producing  the 
desired  resuits.  [AHD  85]  In  SCE,  validation  refers  to  the  process 
of  substantiating  observations  made  about  processes,  using  rules 
for  confirming  observations  defined  in  the  method.  A  valid 
observation  is  one  that  is  accurate  and  has  been  agreed  to  by  the 
team  through  a  consensus  process.  Validated  observations  are 
equivaient  to  findings  after  the  team  concludes  that  data  coverage 
and  corroboration  rules  have  been  met.  The  rationale  for  validation 
is  related  to  the  data  element  objective  of  an  SCE,  obtaining  an 
accurate  picture  of  process  capability  at  a  site. 

Verifying  implementation:  One  of  five  common  features  in  the 
CMM  for  Software.  This  common  feature  describes  the  steps  to 
ensure  that  the  activities  are  performed  in  compliance  with  the 
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process  that  has  been  established.  Verifying  implementation 
typically  encompasses  the  features  of  reviews  and  audits  by 
management,  quality  assurance,  and  other  support  units. 

Weakness:  Indicates  the  team  judgment  that  an  effective 
implementation  of  a  component  of  the  reference  model  is  not 
institutionalized.  In  SCE,  a  weakness  indicates  a  particular  part  of 
the  organization’s  capability  that  has  characteristics  that  increase 
the  risks  due  to  process. 

Ineffective  implementation  of  or  lack  of  practices  which,  in  an 
appraisal  team's  judgment,  interfere  with  effective  performance  of 
software  development  tasks.  CMM  related  weaknesses  are  an 
ineffective  implementation  or  lack  of  implementation  of  one  or  more 
CMM  key  practices  with  no  acceptable  alternative  practices  in 
place.  [Masters  95] 


CMU/SEI-95-TR-012 


©  1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


119 


Glossary 


April  1996 


120  ©  1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


CMU/SEI-95-TR-012 


April  1996 


SCE  Implementation  Examples 


Appendix  B  SCE  Implementation  Examples 

This  appendix  contains  representative  examples  of  RFP  text  and  evalua¬ 
tion  standards  from  the  U.S.  Air  Force  (USAF),  U.S.  Navy  (USN)  and  the 
U.S.  Army  (USA).  Some  other  excerpts  from  recent  RFPs  and  published 
documents  have  also  been  provided.  Here  specific  agency/organizational 
references  have  been  removed.  These  last  example  excerpts  are  provided 
to  illustrate  the  latest  trends  within  the  community  of  SCE  implementation 
within  acquisitions. 

B.1  Using  SCE  Results  in  Air  Force  Source  Selection 

The  first  example  explores  using  the  findings  from  an  SCE  site  visit  in  the 
final  decision  of  a  USAF  source  selection.  The  example  RFP  language  is 
current,  the  data  depicted  and  discussed  has  been  edited  and  does  not 
represent  any  particular  solicitation  past,  present  or  future.  Section  M  of 
the  RFP  notifies  offerors  that  the  use  of  SCE  would  be  evaluated  as  a  spe¬ 
cific  criterion.  Included  in  this  section  will  be  an  example  using  a  color  code 
scheme  as  the  rating  tool  in  the  source  selection  process.  The  discussions 
that  follow,  while  using  data  from  real  acquisitions,  have  been  edited  to 
eliminate  source  selection  sensitivity  or  to  illustrate  a  key  point  about  SCE 
implementation.  A  reference  to  the  PRAG  is  included  at  the  end  of  this  sec¬ 
tion.  SCE  findings  would  be  incorporated  into  the  performance  risk  assess¬ 
ment  report/briefing  if  SCE  is  used  as  part  of  PRAG  activities. 

There  is  a  significant  difference  between  specific  criteria  and  performance 
risk  assessments.  The  source  selection-related  regulations,  regardless  of 
the  implementing  agency,  require  that  specific  criteria  encompass  the 
characteristics  of  the  program  being  acquired.  All  acquisition  agencies  re¬ 
quire  the  establishment  of  order  of  precedence  for  the  various  specific  cri¬ 
teria,  so  that  the  offerors  understand  their  relative  importance  and  can  craft 
their  proposal  accordingly. 

Additionally,  pre-established  standards  of  evaluation  are  prepared  for 
each  criterion  and  the  offerors’  proposal  is  measured  against  those  stan¬ 
dards  by  the  SSEB.  This  evaluation  against  the  evaluation  standards  then 
forms  the  basis  of  comparison  of  one  proposal  to  another,  which  is  done 
in  a  source  selection,  typically  by  a  more  senior  body,  such  as  the  Source 
Selection  Advisory  Council. 

Note  that  in  developing  any  evaluation  standards  (Figure  B-2  and  Figure 
B-3),  the  appropriate  procurement  regulations  should  be  followed  as  well 
as  consulting  and  working  with  the  source  selection  staffs. 
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To  get  the  most  emphasis  of  SCE  use  in  source  selection,  SCE  should  be 
used  as  a  specific  criterion  and  may  also  be  evaluated  by  the  PRAG  for 
performance  risk.  Use  of  SCE  results  as  specific  criterion  and/or  in  the 
PRAG  for  performance  risk  will  be  decided  by  the  SSA  at  the  same  time 
the  SSP  is  approved,  based  on  source  selection  regulations  and  program 
requirements. 

B.1 .1  Usincj  SCE  as  a  Specific  Criterion  for  Award 

Each  offeror’s  proposal  will  be  evaluated  against  the  following  areas  listed 
in  descending  order  of  importance  (list  areas  in  descending  order  of  impor¬ 
tance  or  specify  relative  importance.  Note:  Areas  should  be  limited  to  two 
[including  cost/price],  when  feasible). 

The  technical  area  (or  each  of  the  areas  [except  cost/price]  if  more  than 
two  areas  used)  will  be  rated  in  three  ways: 

•  a  color/adjectival  rating 

•  a  proposal  risk  rating 

•  a  performance  risk  rating 

The  color/adjectival  rating  depicts  how  well  the  offeror’s  proposal  meets 
the  evaluation  standards  and  solicitation  requirements.  Proposal  risk  as¬ 
sesses  the  risk  associated  with  the  offeror’s  proposed  approach  as  it  re¬ 
lates  to  accomplishing  the  requirements  of  this  solicitation.  Performance 
risk  assesses  the  probability  of  the  offeror  successfully  accomplishing  the 
proposed  effort  based  on  the  offeror’s  demonstrated  present  and  past  per¬ 
formance.  The  government  will  conduct  a  performance  risk  assessment 
based  on  the  offeror’s  relevant  present  and  past  performance.  In  assess¬ 
ing  this  risk,  the  government  will  use  performance  data  to  evaluate  the  ar¬ 
eas  listed  above. 

Offerors  are  to  note  that  in  conducting  the  performance  risk  assessment, 
the  government  will  use  both  data  provided  by  the  offeror  and  data  ob¬ 
tained  from  other  sources.  Within  each  area  (other  than  cost/price),  each 
of  the  three  ratings — color/adjectival,  proposal  risk,  and  performance 
risk— will  be  considered  in  making  an  integrated  source  selection  decision 
as  to  which  proposal  is  most  advantageous  to  the  government. 

SCE  should  be  used  as  an  item  under  an  area  of  specific  criterion  such  as 
Technical/Management  and/or  in  the  PRAG  for  performance  risk  assess¬ 
ments.  Ultimately,  how  SCE  findings  are  translated  into  SCE  results  and 
used  in  the  Source  Selection  (SS)  should  be  determined  by  the  SSA  based 
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on  source  selection  regulations  and  program  requirements.  Figure  1  pro¬ 
vides  an  illustration  from  an  acquisition  employing  SCE  as  a  technical  item 
(software  engineering  capability)  in  the  technical  area. 


1  Technical  Area 

Description  I 

Software  Engineering  Capability 
Item 

The  acquisition  organization  will  evaluate  the  offeror’s  software 
process  by  reviewing  its  Software  Process  Improvement  Plan  (SPIP) 
and  by  conducting  an  on-site  visit  using  the  Software  Engineering 
Institute-  (SEI)  developed  Software  Capability  Evaluation  (SCE) 

Method. 

Technical  Approach  item 

The  government  will  evaluate  the  offeror*s  technical  approach  to 
accomplishing  the...  tasks.  The  evaluation  will  assess  the  extent  that 
the  approach  satisfies  the  objective,  goals,  and  requirements  of  the 
Statement  of  Work. 

Management  Item 

The  acquisition  organization  will  evaluate  the  offeror’s  management 
approach  to  accomplish  contract  goals  and  the  extent  to  which  the 
approach  achieves  the  objectives,  goals,  and  requirements  of  the 
solicitation.  The  government  will  focus  on... 

Figure  B-1 :  Sample  Set  of  Specific  Criteria  or  Technical  Items 

What  follows  is  an  example  using  SCE  as  a  specific  criterion  in  making  the 
source  selection  decision.  The  specific  needs  of  the  acquisition  should  dic¬ 
tate  the  exact  approach  to  be  used.  In  this  example,  the  items  of  the  tech¬ 
nical  area  are  listed  ip  descending  order  of  importance.  This  example  is  but 
one  approach  and  method  for  implementing  SCE  findings  in  the  source  se¬ 
lection  decision. 

This  example  continues  the  discussion  of  SCE  as  a  specific  criterion  as  de¬ 
picted  in  Figure  B-1.  The  example  will  illustrate  the  incorporation  of  the 
SCE  findings  into  the  various  source  selection  evaluation  tools/documents 
that  are  used  for  the  source  selection  as  well  as  the  definitions  established 
for  the  various  color  ratings  and  the  identification  of  risk. 

Color  Coding  the  Applying  color  codes  begins  when  the  SCE  team  has  completed  all  site 

Technical  Items  vjgjts  and  the  evaluations  of  the  offerors’  Software  Process  Improvement 

Plans  (SPIP).  In  this  example  the  SPIP  was  requested  to  be  prepared  and 
submitted  separately  at  the  same  time  the  proposal  was  submitted. 

Using  this  approach,  each  technical  item  is  assigned  a  color  which  corre¬ 
sponds  to  a  rating — ^from  “exceptional”  to  “unacceptable.”  For  each  item, 
an  evaluation  standard  is  written  to  define  precisely  what  an  offeror  must 
do  to  be  assigned  a  certain  color. 
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Figure  B-2  shows  how  colors  were  used  and  how  ratings  such  as  “excep¬ 
tional”  were  defined  [USAF  88]. 


Rating 


Blue  (B)  Exceptional  Exceeds  specified  performance  of  capability  in  a 

beneficial  way  to  the  government.  Has  high  probability  of 
satisfying  the  requirement.  Has  no  significant  weakness. 


Green  (G) 

Acceptable 

Meets  evaluation  standards.  Has  good  probability  of 
satisfying  the  requirement.  Any  weakness  can  be  readily 
corrected. 

Yellow  (Y) 

Marginal 

Fails  to  meet  evaluation  standards  or  has  low  probability 
of  meeting  the  requirement;  or  has  significant  but 
correctable  deficiencies. 

Red  (R) 

Unacceptable 

Fails  to  meet  a  minimum  requirement.  Deficiency 
requires  a  major  revision  to  correct. 

Color 


Definition 


Figure  B-2:  Description  of  Colors 


Along  with  each  color,  the  evaluation  team  assigns  a  risk  rating  which  re¬ 
flects  the  risk  associated  with  the  offeror  performing  on  time,  within  budget, 
and  within  the  specified  performance  parameters.  Figure  B-3  lists  the  rat¬ 
ings  and  their  definitions.  This  example  used  the  consistency  between  the 
SCE  findings  and  the  achievability  of  the  offeror’s  software  process  im¬ 
provement  program  to  denote  the  risk  of  the  item.  Software  Engineering 
Capability. 


Risk  Definition 


High  (H)  Likely  to  cause  significant  serious  disruption  of  schedule.  Increase  in  cost,  or 

degradation  of  performance  even  with  special  contractor  emphasis  and  close 
government  monitoring. 

Moderate  (M)  Can  potentially  cause  some  disruption  of  schedule,  increase  in  cost,  or  degradation 

of  performance.  However,  special  contractor  emphasis  and  close  government 
monitoring  will  probably  be  able  to  overcome  difficulties. 

Low  (L)  Has  little  potential  to  cause  disruption  of  schedule,  increase  in  cost,  or  degradation  of 

performance.  Normal  contractor  emphasis  and  government  monitoring  will  probably 
be  able  to  overcome  difficulties. 


Figure  B-3:  Description  of  Risks 

A  complete  set  of  offeror  findings  (strengths  and  weaknesses)  measured 
against  the  CMM  KPAs  should  be  used  in  assigning  color  codes  and  risks. 
The  SCE  team  should  provide  the  SSEB  with  these  findings.  See  Figure 
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B-4,  Figure  B-5,  and  Figure  B-6  for  an  example.  (Figure  B-4  is  a  summary 
of  the  findings,  while  Figure  B-5  and  Figure  B-6  show  the  details  of  that 
summary.) 


Summary  Results 

Strong 

•  Requirements  Management 

•  Peer  Reviews 

•  Software  Project  Tracking  and  Oversight 

Acceptable 

•  Software  Project  Planning 

•  Software  Configuration  Management 

•  Software  Quality  Assurance 

•  Training  Program 

Weak 

•  Organization  Process  Focus 


Figure  B-4:  Summary  Findings  From  a  Recent  SCE 


The  source  selection  organization  should  at  no  time  ask  for  or  accept  find¬ 
ings  from  a  Software  Process  Assessment  (SPA).  As  discussed  in  Chapter 
1 ,  SPA  findings  are  determined  for  a  different  purpose  and  are  inappropri¬ 
ate  for  use  with  SCE  findings  in  a  source  selection  decision. 

The  summary  findings  shown  in  Figure  B-4  reveal  that  only  one  key  pro¬ 
cess  area  was  weak.  The  weaknesses  contributing  to  that  determination 
can  be  found  in  Figure  B-5  and  Figure  B-6.  Although  there  were  weakness¬ 
es  in  other  key  process  areas,  only  the  Organization  Process  Focus  weak¬ 
nesses  were  found  to  be  significant  enough  for  that  KPA  to  be  included  in 
the  summary  findings  weak  area.  The  details  of  that  determination  are 
made  by  the  SCE  team  in  the  context  of  this  specific  acquisition.  This 
means  that  the  SCE  team  used  their  individual  professional  judgment  to 
determine  the  degree  of  satisfaction  of  the  goals  of  each  KPA.  The  context 
of  these  determinations  is  critical  to  the  findings.  For  example,  it  is  possible 
that  an  organization  may  have  a  software  configuration  management  sys¬ 
tem  that  most  experienced  personnel  would  consider  excellent.  However, 
the  SCE  team  may  have  found  that  one  project  does  not  use  it,  another 
project  uses  it  very  effectively,  and  a  third  or  fourth  project  may  use  it  in 
differing  levels  of  application.  This  is  an  example  where  the  SCE  team 
would  be  faced  with  determining,  from  the  organizational  standpoint. 
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whether  a  finding  for  the  Software  Configuration  Management  KPA  is  ac¬ 
ceptable,  weak  or  strong.  On  one  hand  it  was  determined  that  the  config¬ 
uration  management  system  in  place  is  excellent  (a  strength),  on  the  other 
hand  the  evidence  suggests  spotty  implementation  and  or  application  (ac¬ 
ceptable  or  weak?).  Does  this  mean  the  finding  is  reported  as  a  strength, 
acceptable  or  as  a  weakness?  This  type  of  dilemma  is  typical  of  those 
faced  by  the  SCE  team  for  which  the  various  background  experience  in  the 
different  disciplines  comes  into  play  in  providing  consensus  from  a  profes¬ 
sional  judgment  standpoint  on  specific  findings  for  each  KPA  investigated. 


Requirements  Management 
Strengths 

•  Effective  revlew/statusing  mechanism  in  place 

•  Very  strong,  clear  lines  of  authority 

•  Software  engineering  process  represented 
throughout  system  engineering  process  (and 

1  vice  versa) 

I  •  Action  items  tracked  to  closure  by  management 
j  •  Sure  technical  presence  at  managerial  level 

I  Weaknesses 


Improvement  Activities 

none  noted 


Software  Project  Tracking  and  Oversight 
I  Strengths 

I  •  Provides  wide  coverage  of  software  process  at 
j  a  detailed  level 

•  Extensive  use  of  programmers’  notebooks  to 
guide  staff  through  various  phases  of  the  pro¬ 
cess 

j  •  Emphasis  on  populating  useful  software  devel¬ 
opment  folders 

Weaknesses 

•  Lack  of  organizational  consistency 

Improvement  Activities 

none  noted 


Peer  Reviews 

Strengths 

•  Multiple,  rigorous  requirements,  design,  and 
code  inspections  conducted 

•  Training  required  to  participate  on  peer  reviews 

•  Experienced,  senior  people  lead  reviews 

•  Currently  tracing  defects  and  beginning  to 
analyze  results 

Weaknesses 

•  Lack  of  organizational  consistency  in  the 
reviews  of  each  phase 


Improvement  Activities 
none  noted 


Software  Quality  Assurance 
Strengths 

•  Experienced  personnel 

•  Very  good  relationship  with  development 
personnel 

•  Independent  reporting  chain 
Weaknesses 

•  Lack  of  sampling  mechanism 

•  Lack  of  Independent  audit  coverage  and  depth 

•  Resources  lacking  on  some  projects 

Improvement  Activities 

•  Establishing  an  Independent  reporting  chain 

•  Interviewing  for  SQA  personnel 


Figure  B-5:  Detailed  Findings 
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Figure  B-6:  Detailed  Findings  (continued) 


Another  aspect  of  using  SCE  is  illustrated  by  the  use  of  PFNs  to  commu¬ 
nicate  software  process  weaknesses  identified  by  SCE  to  the  offerors  with¬ 
in  the  competitive  range.  A  Clarification  Request  (CR)  should  be  used  to 
communicate  a  weakness  initially.  A  Point  for  Negotiation  (PFN)  can  be 
used  to  identify  those  points  the  government  wishes  to  discuss  further.  A 
PFN  or  CR  will  never  be  used  to  identify  a  deficiency.  The  SSEB  then  con¬ 
siders  their  responses  with  the  original  SCE  findings  before  making  a  final 
determination  against  the  evaluation  standard.  This  approach  allows  the 
offerors  the  opportunity  to  point  out  any  oversights  on  the  part  of  the  SCE 
team.  The  SCE  team  could  prepare  a  PFN  (or  CR  if  appropriate)  to  let  of- 
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ferors  know  what  weaknesses  were  found.  Figure  7  is  an  example  of  a 
PFN.  This  example  details  the  specific  weaknesses  found  by  the  SCE 
team  that  made  the  KPA  Organization  Process  Focus  weak. 


Source  Selection  Information  (See  FAR  3.104) 
For  Official  Use  Only  (when  filled  in) 


POINT  FOR  NEGOTIATION 


Government  Reference: 
IFPP  Paragraph  3.3.4 


Offeror: 

XYZ  Corporation 


Offeror  Reference: 


Register  Number: 
PFN-XYZ-S-001 


The  key  process  area  (KPA)  found  by  the  Software  Capability  Evaluation 
(SCE)  to  be  weak  is  Organization  Process  Focus.  The  detailed  findings  leading 
to  this  conclusion  are  as  follows: 

•  Lack  of  buy  in  from  the  engineering  staff  (many  unaware  of  existence) 

•  Lack  of  SEPG  focus  and  record  of  accomplishment 


SSEB/T  Chairperson: 


Source  Selection  Information  (See  FAR  3.104) 
For  Official  Use  Only  (when  filled  in) 


Figure  B-7:  Findings  Incorporated  Into  a  Point  For  Negotiation 


The  findings  that  go  into  a  PFN  may  vary.  One  acquisition  organization’s 
approach  was  to  provide  only  those  weaknesses  in  the  PFN  that  caused 
an  entire  key  process  area  to  be  evaluated  as  weak  (as  in  Figure  B-7). 
Those  are  significant  weaknesses  which,  depending  upon  the  affected  key 
process  area,  may  influence  the  evaluation  standard  one  way  or  another. 
Alternatively,  the  entire  findings  set  may  be  communicated  in  the  PFN.  In 
deciding  what  to  include  in  the  PFN/CR,  the  SCE  team  leader  should  work 
very  closely  with  the  PCO,  SSEB  chairperson,  and  the  acquisition  organi¬ 
zation’s  legal  advisor. 
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A  PFN/CR  is  a  way  to  communicate  an  SCE  weakness(es)  to  an  offeror 
and  allow  the  offeror  to  respond  with  one  of  the  following: 

•  evidence  showing  the  government’s  SCE  team  made  an  oversight 

•  a  response  accepting  the  findings 

•  a  response  accepting  the  findings  and  identifying  improvement  activi¬ 
ties  to  remedy  the  weaknesses 

•  a  combination  of  the  above  previous  responses 

A  cover  letter  sent  with  the  PFNs/CRs  will  explain  how  the  offeror  may  re¬ 
spond.  It  is  recommended  that  the  letter  include  a  page  limitation  for  the 
offeror’s  response  so  that  the  SSEB  is  provided  with  only  relevant  evi¬ 
dence. 

When  the  responses  to  the  PFNs/CRs  have  been  received  from  the  offer¬ 
ors  (typically  five  to  seven  days  are  allowed  for  responses)  the  SCE  team 
leader  should  analyze  them  to  see  if  material  changes  in  the  findings  are 
required  that  would  necessitate  recalling  the  SCE  team.  The  only  time  the 
SCE  team  would  reverse  a  decision  on  a  finding,  is  if  the  offeror  shows 
proof  that  the  team  overlooked  something. 

The  SCE  team  performs  an  analysis  and  makes  any  final  adjustments  to 
the  findings.  These  findings  will  be  factored  into  the  technical  area/item 
evaluation  results  for  each  offeror.  The  manner  in  which  SCE  findings/re¬ 
sults  are  factored  into  the  source  selection  results  depends  upon  how  SCE 
was  structured  into'  the  source  selection  (e.g.  items,  factors  etc.).  Your 
PCO  and  procurerpent  regulations  will  guide  you  through  this  step.  Figure 
8  and  Figure  9  provide  an  example  item  summary  for  the  set  of  findings 
shown  in  Figure  4,  Figure  5,  and  Figure  6.  The  example  assumes  that  no 
changes  to  the  findings  were  made  during  the  PFN/CR  process. 

The  item  summary  contains  the  color  rating  and  associated  risk  for  the  re¬ 
spective  offer,  some  background  on  the  projects  the  SCE  evaluated,  the 
summary  and  detailed  findings  made  by  the  SCE  team  while  on  site,  and 
a  statement  justifying  the  assigned  risk.  In  order  to  determine  the  color  rat¬ 
ing,  the  SCE  team  applied  the  findings  to  the  evaluation  standard.  Similar¬ 
ly,  for  this  example,  the  risk  was  assigned  based  upon  consistency 
between  the  offeror’s  communicated  capability  found  in  the  SPIP  and  the 
actual  SCE  findings.  In  this  example,  the  offeror  was  rated  blue  with  a  low 
risk.  The  item  summary  then  points  out  the  various  strengths  and  weak¬ 
nesses  in  their  appropriate  location  and  justifies  the  risk  rating. 
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At  this  level  of  evaluation,  within  the  SSEB,  the  offerors  are  only  compared 
to  a  pre-established  standard.  No  offerors  are  compared  to  one  another. 


Source  Selection  Information  (See  FAR  3.104)  —  For  Official  Use  Only  (When  Filled  in) 

Item  Summary 

Area:  Technical  Item:  T3/Software  Offeror:  XYZ  Corp  Color  Rating:  Blue 

Engineering  Capability 

Description  of  Proposal 

The  offeror  proposed  a  software  PIP  which... 

The  software  Process  Improvement  Plan  was  found  to  be  consistent  with  the  SCE  findings.  The  offeror’s 
program  of  software  process  improvement  is  genuine,  with  considerable  emphasis  on  organizational 
standardization  and  removal  of  defects  through  rigorous  reviews.  The  projects  examined  as  part  of  the 
Software  Capability  Evaluation  (SCE)  are  as  follows: 

•  ABCD  •  HAVE  GOLD  PLATE  •  COBRA  LIBRARY  •  CCXY2 


Strengths 

Requirements  Management 

•  Defined  review/status  mechanism  in  place 

•  Very  clear,  strong  lines  of  authority 

•  Software  engineering  represented  throughout  system  engineering  process  (and  vice  versa) 

•  Action  items  tracked  to  closure  by  management 

•  Sure  technical  presence  at  management  Ij^vel 

Software  Project  Tracking  and  Oversight 

•  Provides  wide  coverage  of  software  process  at  a  detailed  level 

•  Extensive  use  of  programmers  notebooks  to  guide  staff  through  phases  of  the  process 

•  Firm  emphasis  on  populating  useful  software  development  folders 

Peer  Reviews 

•  Multiple,  rigorous  requirements,  design,  and  code  inspections  conducted 

•  Training  required  to  participate  on  peer  reviews 

•  Experienced,  senior  people  lead  reviews 

•  Currently  tracking  defects  and  beginning  to  analyze  results 

Item  Chief  Signature:  Area  Chief  Signature: 

Figure  B-8:  Findings  Incorporated  in  Item  Summary 
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Acceptable  Points 

SQA 

•  Experienced  personnel  and  independent  reporting  chain 
Software  Project  Planning 

•  Procedure  for  sizing  and  costing  software  exist  project  to  project 

•  Extensive  collection  of  management  metrics  and  tracking  of  progress 

SCM 

•  Effective  change  control  process  and  traceability  between  development  projects 
Training  Program 

•  Solid  emphasis  from  management  and  extensive  in-house  software  courses 

•  SEPG 

•  An  organizational  function  exists  with  full-time  resources  in  place 

Weaknesses 

The  Key  Process  Area  found  by  the  Software  Capability  Evaluation  to  be  weak  was: 
Organization  Process  Focus 

•  Lack  of  buy-in  from  the  engineering  staff  (many  unaware  of  existence) 

•  Lack  of  SEPG  focus  and  record  of  accomplishment 


Overall  Risk  Assessment  and  Evaluation  Summary 

Low  Risk:  The  consistency  between  their  SCE  findings  and  software  process  improvement  plan  shows 
they  understand  their  current  maturity  level  and  where  they  are  going  as  an  organization.  They  are  very 
strong  technically  (very  close  to  being  strong  in  all  the  key  process  areas)  and  are  committed  to 
developing  quality  software  using  a  continually  improving  development  process.  This  contractor’s 
commitment  to  process  improvement  was  further  evidenced  by  the  process  rigor  In  place  on  one  of  their 
commercial  programs  where  no  development  standards  were  required.  Their  process  was  still  the  same 
and  management  exercised  the  same  controls. 


Figure  B-9:  Findings  Incorporated  in  item  Summary  (continued) 


Item  Summaries  are  reviewed  by  the  SSEB/T  chairperson  and  then  an 
area  summary  is  prepared  which  normally  “rolls  up”  all  (or  most)  of  the 
strengths  and  weaknesses  from  the  individual  item  summaries  and  then 
identifies  an  overall  risk  for  that  area.  This  information  is  reviewed  by  the 
PCO,  legal  representatives,  and  the  SSAC.  The  legal  and  PCO  review  will 
examine  everything  to  insure  that  the  evaluation  standards  have  been  con¬ 
sistently  applied  and  that  the  item  and  area  summaries  contain  consistent 
types  and  levels  of  information.  The  SSEB/T  will  present  this  information 
to  the  SSAC.  The  SSAC  members  will  analyze  the  SSEB/T’s  evaluation 
results  and  start  the  process  of  comparing  each  offerors  strengths,  weak¬ 
nesses  and  risk — an  activity  the  SSEB  is  not  allowed  to  do. 
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In  parallel,  the  SSEB  will  make  a  formal  presentation  to  the  SSAC  outlining 
the  color  codes,  strengths,  weaknesses  and  risks  for  each  offeror  for  each 
item  and  area  resulting  from  their  evaluations.  During  this  presentation,  the 
SCE  team  leader,  as  a  member  of  the  SSEB,  should  be  prepared  to  elab¬ 
orate  on  any  of  the  findings  from  any  of  the  offerors.  For  example,  the  SCE 
team  leader  should  be  prepared  to  explain  not  only  why  an  offeror  was 
weak  in  software  configuration  management,  but  also  why  the  SCE  team 
found  their  change  control  process  lacking.  The  SSAC  will  want  to  ensure 
that  the  SSEB  can  substantiate  their  findings  with  documented  evidence. 

At  this  point  in  the  source  selection  process,  the  SSAC,  after  completing 
their  comparative  analysis  of  all  final  proposals’  strengths,  weaknesses 
and  risks,  may  elect  to  assign  a  different  color  code  separate  from  the 
SSEB  or  It  may  ask  the  SSEB  to  reconsider  its  color  codes  in  light  of  infor¬ 
mation  discussed  in  the  SSAC  briefing.  These  actions  are  normally  done 
on  an  “exception”  basis  and  are  not  common  since  the  SSAC  would  have 
reviewed  this  material  at  the  time  of  competitive  range  and  before  BAFOs 
were  issued;  therefore,  any  “disconnects”  should  be  resolved  before 
BAFOs  are  received.  Unless  an  offeror  completely  changes  its  proposal 
approach,  there  should  be  no  “surprises”  in  the  BAFOs.  The  SSAC  will  en¬ 
sure  that  the  evaluation  for  each  criterion  has  been  consistently  and  fairly 
applied  to  all  offerors. 

Figure  1 0  shows  one  way  the  findings  from  a  series  of  SCEs  has  been  pre¬ 
sented  formally  to  a  SSAC.  Each  offeror’s  technical  rating,  strengths  and 
weaknesses,  risk,  and  a  summary  explaining  the  basis  for  the  risk  are 
identified  and  placed  next  to  the  other  offerors  so  that  the  SSAC  may  com¬ 
pare  and  discuss  them  during  a  presentation.  This  normally  represents  the 
lowest  level  of  detail  presented  to  the  SSAC  during  the  formal  presenta¬ 
tion.  It  is  during  this  presentation  that  an  SCE  team  leader  may  have  to  ar¬ 
ticulate  why  certain  key  process  areas  were  a  strength  or  weakness  for  a 
particular  offeror.  The  expertise  of  the  SCE  team  leader  is  needed  to  com¬ 
municate  why  a  KPA  was  strong  or  weak  and  its  significance  within  the 
software  process. 
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ltem:T-3  Software  Engineering  Capability 

Offeror  A 

Offeror  B 

Offeror  C 

Blue 

Yellow 

Yellow 

Strength 

•  Requirements 

Management 

•  Peer  Reviews 

•  Software  Project  Tracking 
and  oversight 

•  None 

•  Organization  Process 

Focus 

Weakness 

•  Software  Engineering 
Process  Group 

•  Organization  Process 

Focus 

•  Software  Quality 

Assurance 

•  Training  Program 

•  Peer  Reviews 

•  Software  Project  Tracking 
and  Oversight 

•  Training  Program 

Risk 

Offeror  is  very  strong 
technically  and  is  committed 
to  developing  quality 
software  using  a 
continuously  improving 
development  process 

Because  of  the  large 
disparity  between  SCE 
findings  and  their  submitted 
SPIP,  it  is  highly  questionable 
whether  the  software  process 
improvement  is  being 
implemented 

Offer’s  SPIP  indicated  they 
are  at  the  initial  maturity  level 
with  their  best  practices 
being  applied  to  all  new 
programs 

L 

H 

M 

Figure  B-10:  Findings  Output  From  the  Evaiuation  Standard 


The  SCE  written  report  must  also  back  up  and  provide  substantiation  or  ar¬ 
ticulate  reasons  for  the  ratings’  assigned  since  the  briefing  is  reduced  to 
“bullets”  only  and  should  be  derived  from  the  detailed  written  findings. 

Figure  10  illustrates  how  risk  was  assigned  to  the  software  engineering  ca¬ 
pability  technical  score  (color  rating)  in  a  recent  source  selection.  Note  that 
Offerors  B  and  C  have  yellow  as  their  technical  score,  but  Offeror  B  has  a 
high  risk  and  Offeror  C  has  a  moderate  risk;  yet  Offeror  C  has  three  weak 
Key  Process  Areas  and  Offeror  B  has  only  two.  How  did  this  occur? 

Risk  in  this  acquisition  was  assigned  based  upon  the  consistency  of  the  or¬ 
ganization’s  process  improvement  program  and  the  SCE  findings,  be¬ 
cause  it  was  stated  clearly  in  the  RFP  for  this  acquisition  that  an 
organization  could  be  at  the  Initial  maturity  level  and  still  be  awarded  the 
contract.  It  was  also  stated  in  the  Instructions  for  Proposal  Preparation  (IF- 
PP)  that  risk  would  be  used  as  a  measure  of  an  organization’s  process  im¬ 
provement  realism.  If  an  organization  had  a  realistic  program  of  software 
process  improvement,  then  they  were  considered  low  risk,  regardless  of 
their  current  maturity  level  rating.  If  an  offeror  claimed  to  be  at  the  Defined 
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or  Managed  maturity  level  in  its  SPIP,  but  the  SCE  findings  showed  the  of¬ 
feror  to  be  at  the  Initial  or  Repeatable  maturity  level,  then  the  SSEB  would 
assign  either  a  high  or  moderate  risk.  This  assignment  depended  upon  the 
magnitude  of  the  disparity  between  the  SPIP  and  the  SCE  findings. 

Offeror  B  had  identified  itself  as  being  at  the  Defined  maturity  level  and  did 
not  have  an  improvement  plan  that  would  substantiate  its  progress  through 
the  lower  maturity  level.  The  SSEB/SCE  team  determined  Offeror  B  to  be 
closer  to  the  Initial  maturity  level.  In  short,  Offeror  B  was  unaware  of  its  ac¬ 
tual  lower  maturity  level  and  was  consequently  assigned  a  high  risk  with 
only  two  weak  key  process  areas,  while  Offeror  C  received  a  moderate  risk 
with  three.  Offeror  C,  on  the  other  hand,  had  a  realistic  SPIP  indicating  it 
was  at  the  Initial  maturity  level  with  its  best  practices  being  applied  to  all 
new  programs.  The  SCE  findings  confirmed  this  and  resulted  in  assigning 
a  lower  risk  to  this  offeror. 


Area:  Technical 

Off¬ 
eror  A 

Off¬ 
eror  B 

Off¬ 
eror  C 

Software  Engineering 
Capability 

Color 

Blue 

Yellow 

Yellow 

Risk 

L 

H 

M 

Technical  App. 

Color 

Green 

Yellow 

Blue 

Risk 

L 

L 

L 

Management 

Color 

Green 

Green 

Blue 

1 

Risk 

L 

L 

L 

SUMMARY  RESULTS 

Color 

Green 

Yellow 

Green 

Risk 

L 

H 

M 

Figure  B-1 1 ;  Technical  Area  Summary 


The  last  step  of  the  process  is  the  integration  of  the  SCE  technical  rating 
and  risk  factor  with  those  of  the  other  technical  items  to  produce  a  techni¬ 
cal  area  summary,  as  shown  in  Figure  1 1 .  At  this  point,  the  SSAC  will  in¬ 
tegrate  the  color  codes  and  risk  factors  into  area  summaries  based  upon 
their  own  analysis  and  presents  them  to  the  SSA.  The  SSAC  then  con¬ 
ducts  a  comparative  analysis  of  all  offerors’  strengths,  weaknesses  and 
risks  as  presented  by  the  SSEB/T  on  these  item  and  area  summaries  and 
presents  it  to  the  SSA.  Note:  SSAC  does  not  make  written  recommenda- 
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tions  to  the  SSA.  Note  that  in  this  example  the  items  in  the  Technical  Area, 
Management,  Technical  Approach  and  Software  Engineering  Capability 
are  listed  in  descending  order  of  importance.  This  illustration  of  risk  identi¬ 
fication  and  assessment  is  not  the  sole  method  for  approaching  the  risk 
problem.  Acquisitions  should  tailor  the  risk  assignment  to  the  specific 
needs  of  the  acquisition. 

Performance  Risk  Offerors  past  and  present  performance  is  evaluated  by  the  PRAG.  Their 

Analysis  Group  results  will  be  presented  to  the  SSA  in  the  form  of  performance  risk. 

(PRAG) 

Performance  Risk  Assessment  Definitions: 

High:  Significant  doubt  exists,  based  on  offeror’s  performance  records, 
that  the  offeror  can  perform  the  proposed  effort. 

Moderate:  Some  doubt  exists,  based  on  the  offeror’s  performance  records, 
that  the  offeror  can  perform  the  proposed  effort. 

Low.  Little  doubt  exists,  based  on  the  offeror’s  performance  records,  that 
the  offeror  can  perform  the  proposed  effort. 

N/A:  No  performance  record  identifiable. 

B.2  United  States  Navy  SCE  Implementation  Example 

The  following  example  is  representative  of  United  States  Navy  acquisition 
organizations  which  use  an  algorithmic  approach  to  source  selection, 
where  percentages  are  allocated  to  the  various  specific  criteria  and  points 
can  be  earned  for  each  criterion.  Typically  percentage  points  are  distribut¬ 
ed  among  the  various  criterion.  At  a  high  level,  typical  criterion  of  Techni¬ 
cal,  Management  and  Cost  would  have  different  percentages  of  weighting 
or  raw  points  (e.g.  Technical  45%,  Management  30%  and  Cost  25%).  Note 
that  nominally  these  criterion  are  further  broken  down  into  lectors  or 
items”  that  further  distribute  points.  For  example  the  overall  evaluation  cri¬ 
terion  Technical  could  be  subdivided  into:  Technical  Approach,  Networks, 
Software  Engineering,  and  Reuse  Initiative.  The  individual  solicitation 
(RFP)  requirements  will  determine  the  specific  set  of  evaluation  criteria. 
The  allocated  percentage/points  would  then  be  allocated  among  those 
four  factors.The  points  are  then  totaled  according  to  the  algorithm.  Part  of 
the  algorithm  includes  allocating  percentages  of  the  source  selection  deci¬ 
sion  to  the  cost  and  management  areas. 
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Note  that  the  various  sections  of  the  RFP  (Sections  H,  L,  M,  Figure  B-12, 
Figure  B-13,  Figure  B-14,  Figure  B-15)  aithough  slightiy  different  in  actual 
text  from  that  shown  in  Part  2  of  this  document,  all  provide  explicit  refer¬ 
ence  to  using  the  CMU/SEI-93-TR-24  “Capability  Maturity  Model  for  Soft- 
wareT  and  CMU/SEI-TR-25  “Key  Practices  of  the  Capability  Maturity 
Mode\”  as  the  reference  model  for  use  during  onsite  reviews  (SCE  con¬ 
duct)  at  contractor  sites. 


section  I 


|a  major  emphasis  at  <Navy  Organization>  and  of  this  contract  is  software  engineering  and  quality 
Improvement.  It  is  mandatory  that  offerors  demonstrate  they  have  plans  in  place  to  improve  the  quality 
|and  productivity  for  software,  are  progressing  toward  their  Improvement  goals,  and  continue  to  improve 
jtheir  software  maturity  level  throughout  the  life  of  this  contract.  During  contract  performance  the 
iGovernment  reserves  the  right  to  conduct  on-site  reviews  of  the  contractor’s  software  engineering 
Iprocesses.  Key  Process  Areas  as  defined  in  “Capability  Maturity  Model  for  Software,  Version  1.1, 
|February  1993,  CMU/SEI/93-TR-24/25”  may  be  reviewed. 

The  methodologies,  procedures  and  techniques  for  software  engineering  described  by  the  offeror  in  thelr| 
pproposal  shall  be  binding  requirements  upon  the  offeror  in  the  performance  of  all  software  work  under 
[this  contract. 


Figure  B-12:  USN  RFP  Section  H 
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It  is  mandatory  that  offerors  demonstrate  they  have  plans  in  place  to  improve  the  qual-| 
pty  and  productivity  for  software,  are  progressing  toward  their  improvement  goals,  and 
continue  to  improve  their  software  maturity  level  throughout  the  life  of  this  contract. 
|Offerors  shall  submit  as  part  of  their  proposal  the  following: 

1.  Software  Process  Improvement  Plan  -  This  plan  should  describe  the  offeror’s 
Iplans  and  schedule  for  improving  their  current  software  engineering  processes/prac- 
|tices,  as  they  relate  to  the  Key  Process  Areas. 

2.  Company  Software  Standards/Practices  documentation  -  This  documenta¬ 
tion  shall  address  methods  in  which  software  engineering  work  shall  be  conducted 
iunder  the  contract.  The  offeror  shall  provided  evidence  of  documented  standards  and 
Ipractices.  Within  the  Software  Standards/Practices  documentation,  each  offeror  shall 
Idetail  the  extend  of  their  employment  of  standardized,  state  of  the  art  software  stan- 
Idards  and  practices.  The  above  information  will  be  used  to  evaluate  the  Software  Engi-| 
ineering  evaluation  criterion  (see  Section  M). 


|An  on-site  evaluation  to  compare  the  information  submitted  with  the  proposal  (i.e. 
|Software  Process  Improvement  Plan  and  the  Software  Standards/Practices  documenta-! 
|tion)  to  current  processes/practices  may  be  performed  (see  Section  M).  During  the  on-N 
|site  evaluation,  information  provided  in  the  proposal  will  be  validated  through  inter¬ 
views  with  various  personnel  and  thrbugh  documentation  reviews  for  the  projects 
3eing  evaluated.  Offerors  should  have  available  the  software  managers  and  other  key 
personnel  for  the  projects  as  well  as  the  documentation  necessary  to  support  the  evalu-| 
|ation.  The  on-site  evaluation  will  focus  on  the  offeror’s  compliance  with  the  Key  Pro¬ 
cess  Areas.  The  Key  Process  Areas  along  with  page  references  to  “Capability  Maturity 
|Model  for  Software,  Version  1.1,  February  1993,  CMU/SEI/93-TR-24/25’’,  which  will 
56  furnished  upon  request,  are  presented  below.  Page  references  provide  the  general 
iscope  of  inquiry  concerning  the  Key  Process  Areas. 


Figure  B-1 3:  USN  RFP  Section  L 
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iSection  L  (continued) 
iKey  Process  Areas: 

iRequirements  Management,  Pages  L2-1  through  L2-10 
iSoftware  Project  Planning,  PagesL2-l  1  through  L2-28 
ISoftware  Project  Tracking  and  Oversight,  Pages  L2-29  through  L2-42 
iSoftware  Subcontractor  Management,  Pages  L2-43  through  L2-58 
ISoftware  Quality  Assurance,  Pages  L2-59  through  L2-70 
iSoftware  Configuration  Management,  Pages  L2-71  through  L2-84 
iOrganization  Process  Focus,  Pages  L3-1  through  L3-10 
lOrganization  Process  Definition,  Pages  L3-1 1  through  L3-24 
iTraining  Program,  Pages  L3-25  through  L3-36 
pintegrated  Software  Management,  Pages  L3-37  through  L3-58 
ISoftware  Product  Engineering,  Pages  L3-59  through  L3-82 
iintergroup  Coordination,  Pages  L3-83  through  L3-92 
pPeer  Reviews,  Pages  L3-93  through  L3-100 

lOfferors  shall  submit  with  their  proposal  the  following: 

I  1 .  Project  Profiles  (enclosure  (2)) 

I  2.  Software  Process  Maturity  Questionnaires  (enclosure  (1)) 

I  3.  Corporate  organization  charts  showing  complete  administrative  and  func- 
Itional  interfaces  for  the  projects  being  evaluated. 


phis  information  will  not  be  evaluated  unless  an  on-site  evaluation  is  performed. 

(However,  failure  to  submit  this  information  could  result  in  an  offeror  receiving  a 
significantly  reduced  score  in  the  software  engineering  criterion  in  the  event  an  on¬ 
site  evaluation  is  performed.  This  above  information  will  be  used  to  prepare  for  the 
on-site  evaluation  in  the  event  that  the  evaluation  is  preformed,  (see  Section  M). 


iirhe  project  profiles  and  Software  Process  Maturity  Questionnaires  shall  be  submit- 
ited  for  four  projects.  These  projects  must  have  been  completed  within  the  last  year 
lor  are  currently  in  progress  and  are  similar  in  scope  and  magnitude  to  the  proposed 
jcontract.  Responses  shall  be  submitted  on  the  Software  Process  Maturity  Question- 
inaires  per  instructions  in  enclosure  (1).  All  questions  must  be  answered,  or  docu¬ 
mentation  in  the  “comments”  space  of  the  Software  Process  Maturity 
Questionnaire. 
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Figure  B-14;  USN  RFP  Section  L  (continued) 

Figure  B-15,  Section  M  of  the  RFP,  identifies  for  the  contractor  community 
what  ls  evaluated  not  the  specifics  of  exactly  how  the  criteria  is  scored. 

Figure  B-16  is  an  example  evaluation  standard  demonstrating  a  numerical 
approach  to  scoring  the  software  engineering  capability  specific  criterion. 
Note  that  it  is  an  internal  document  that  according  to  the  organization’s  in¬ 
ternal  acquisition  and  procurement  regulations  would  be  treated  as 
“source  selection  sensitive”  and  not  published  for  the  contractor  communi¬ 
ty’s  information. 


Section  M 

tfhe  technical  score  for  the  Software  Engineering  capability  evaluation  criterion  shall  be  determinecj 
hrough  an  evaluation  of  the  offeror’s  strengths  and  weaknesses  in  Key  Process  Areas  as  measured  b^ 
he  Software  Process  Improvement  Plan  and  the  Software  Standards/Practices  documentation  that  are 
provided  with  the  proposal.  An  on-site  evaluation  to  compare  proposed  processes/practices  may  be  perl 
jpormed.  If  this  on-site  evaluation  is  performed,  all  acceptable  offerors  will  be  evaluated.  This  evaluatiorl 
bould  result  in  reduced  technical  scores  if  information  provided  in  the  proposal  is  found  to  be  incorrect  of 
not  verifiable  with  the  Project  Profiles,  Software  Process  Maturity  Questionnaires  and  corporate  organi| 
ation  charts  provided  with  the  offeror’s  proposal. 


’15:  USN  Section  M  Evaluation  Criteria 


) 
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(Description:  Software  Engineering  Capability:  Evaluation  will  be  made  of  the  offeror’s  organizational  soft 
Jware  process  capability  using  Software  Capability  Evaluation  (SCE),  their  program  for  software  proces 
jimprovement,  and  the  extent  to  which  their  software  process  supports  the  goals,  objectives,  and  require 
Iments  of  the  solicitation.  A  total  of  28  points  can  be  earned  from  this  specific  criterion. 


jStandard: 

1)  The  Software  Process  Improvement  Plan  (SPIP)  submitted  with  their  proposal  realistically  portray 
their  current  process  capability  and  presents  a  realistic  plan  to  process  improvement.  The  offeror’ 
plan  Is  consistent  with  their  SCE  findings.  The  SPIP  outlines  the  offeror’s  plan  to  achieve  higher  ma 
turity  levels  and  demonstrates  the  offeror  understands  software  process  Improvement,  both  techni 
cally  and  In  the  effort  required  to  Increase  and  sustain  higher  maturity.  The  offeror,  shall  earn  zer 
points  If  the  plan  is  unrealistic  or  Inconsistent  with  the  SCE  findings.  The  offeror  shall  earn  up  to  tw 
points  if  the  plan  is  realistic  and  correlates  with  the  SCE  findings. 

)  For  each  of  the  following  key  process  areas,  the  offeror  earns  a  point  for  each  key  process  area  foun 
to  be  strong  or  acceptable  In  the  SCE  findings: 

•  Software  Configuration  Management 

•  Software  Quality  Assurance 

•  Software  Subcontract  management 

•  Software  Project  Tracking  and  Oversight 

•  Software  Project  Planning 

•  Requirements  Management 

3)  For  the  offeror  to  earn  any  points  for  the  following  key  process  areas,  the  offeror  must  have  bee 
strong  or  acceptable  in  all  the  key  process  areas  identified  in  (2)  and  have  earned  at  least  one  poin 
I  for  criterion  (1 ).  If  those  criteria  have  been  met,  then  the  offeror  earns  an  additional  point  for  each  o 
5  the  following  key  process  areas  found  to  be  strong  or  acceptable  in  the  SCE  findings: 

i  •  Peer  Reviews  / 

•  Intergroup  Coordination 

•  Software  Product  Engineering 

•  Integrated  Software  Management 

•  Training  Program 
S  •  Organization  Process  Definition 

•  Organization  Process  Focus 


Figure  B-16:  USN  Numerical  SCE  Evaluation  Standard 


B.2.1  Applying  Percentages  and  Points  Among  Evaluation  Criteria 

This  approach  integrates  the  SCE  KPA  findings  numerically.  In  this  exam¬ 
ple  five  representative  discrete  evaluation  criteria  could  be: 

1 .  Methodology  for  Providing  Services  Proposed 

2.  Technical  Approach 
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3.  Corporate  Resources/Experience/Management 

4.  Software  Engineering 

5.  Cost 

The  score,  based  on  28  points  out  of  a  grand  total  of  1 00  for  the  entire  eval¬ 
uation,  can  be  adjusted  to  reflect  the  unique  needs  of  a  particular  acquisi¬ 
tion  in  the  same  manner  as  the  color  based  approach.  This  numerical 
approach  is  not  a  scoring  mechanism  to  derive  maturity  levels  or  KPA 
strengths  and  weaknesses.  Instead,  it  is  a  system  for  integrating  findings 
of  strengths,  weaknesses,  and  improvement  activities  into  a  numerically- 
based  source  selection  process.  The  numerical  approach  works  in  a  se¬ 
quential  fashion. 

1 .  The  first  item  in  Figure  B-16  addresses  an  offeror’s  SPIP.  Depending 
upon  the  SPIP  realism  and  its  consistency  with  the  SCE  findings,  the 
offeror  may  earn  up  to  two  points. 

2.  The  second  item  addresses  the  Repeatable  maturity  level  KPAs.  An 
offeror  may  earn  up  to  12  points  for  this  item  depending  on  the  KPA 
ratings  that  are  “Strong”  or  “Acceptable.” 

3.  The  third  item  addresses  the  Defined  maturity  level  KPAs.  A  realistic 
SPIP,  a  “Strong”  or  “Acceptable”  rating  in  each  of  the  KPAs  identified 
in  item  two,  and  “Strong”  or  “Acceptable”  ratings  in  the  Defined  level 
KPAs  may  earn  up  to  14  points. 

This  approach,  whil^  somewhat  stringent,  captures  the  spirit  of  the  CMM 
because  it  emphasi2es  that  the  lower  level  KPAs  must  be  in  place  before 
optimal  benefit  can  be  attained  from  achieving  higher  level  KPAs. 

This  section  presented  examples  of  how  an  evaluation  standard  can  be 
written  which  successfully  de-emphasizes  maturity  levels  while  keeping 
with  the  spirit  of  the  CMM.  Trained  SCE  users  should  be  able  to  take  these 
examples  and  tailor  one  of  them  to  meet  the  specific  needs  of  their  acqui¬ 
sition.  Thus,  SCE  can  contribute  effectively  to  the  source  selection  deci¬ 
sion.  The  findings,  provided  to  the  SSEB  by  the  SCE  Team,  are  a  snapshot 
of  process  capability  for  a  specific  site  at  a  particular  point  in  time.  The  way 
those  findings  are  used  by  the  acquisition  organization  can  be  modified 
through  the  design  of  the  evaluation  standard. 
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B.3  Using  SCE,  Army  Source  Selections 

The  figures  below  show  U.S.  Army  Computer  Electronics  Communications 
(CECOM)  RFP  language.  Note  the  reference  to  the  Software  Process  Risk 
Evaluation  and  the  standard  being  the  SEI  CMM  in  the  RFP  Section  L  Fig¬ 
ure  B-17.  Further  note  the  RFP  Section  M,  Figure  B-19,  stating  the  SPRE 
methodology  is  consistent  with  the  SEI’s  Software  Capability  Evaluation 
methodology. 


L-XX.X  SOFTWARE  PROCESS  RISK  EVALUATION. 


The  Software  Process  Risk  Evaluation  (SPRE),  an  integral  part  of  the  source  selection  process,  deter] 
mines  the  risks  associated  with  the  offeror's  software  development  process.  The  SPRE  determines  riskd 
Ibased  upon  the  strengths  and  weaknesses  of  the  offeror's  software  development  process,  and  the  offeror'^"^^ 
Initiated  or  planned  software  process  improvement  efforts. 


"Software  process  capability  will  be  determined  through  an  evaluation  of  the  offeror's  strengths  and  weak 
inesses.  The  standard  for  the  evaluation  will  be  the  Capability  Maturity  Model,  version  1.1,  documented  ir 
CMU/SEI-93-TR-24  and  CMU/SEI-93-TR-25.  The  results  will  be  an  organizational  composite,  based  or 
responses  to  the  software  process  maturity  questionnaire,  interviews  with  organization  personnel,  and  re¬ 
view  of  organization  documentation  and  artifacts.  The  projects  to  be  evaluated  will  be  selected  by  the  gov- 
|ernment  from  the  submitted  project  profiles.  A  risk  assessment,  evaluating  current  process  practices  as 
^ell  as  proposed  processes  improvements,  will  be  performed.  The  SPRE  shall  require  the  following: 

a.  The  government  reserves  the  right  to  perform  an  SPRE  on  each  organization  involved  in  the  devel 
opment  effort.  The  SPREs  of  all  organizations  evaluated  will  be  considered  part  of  the  overall  SPRE 
risk  rating  for  the  offeror.  For  the  purposes  of  this  a  nd  all  the  following  bullets,  an  organization  is  definec 
as  each  geographically  separate  corporation,  division,  or  site.  Therefore,  each  organization  should  pre¬ 
pare  for  a  maximum  of  a  five  day  on-site  evaluation  period,  at  a  time  to  be  determined  by  the  govern 
ment. 

b.  Each  organization  should  prepare  to  identify  by  name,  and  provide  access  to  the  following  personne 
for  on-site  evaluation: 

-  -  A  corporate  officer  or  senior  manager  for  the  organization  being  evaluated 

-  -  Project  manager  and  software  manager  for  each  project  subject  to  the  evaluation 

-  -  Software  Quality  Assurance  representative  for  each  project  subject  to  evaluation 

-  -  Testing  representatives  for  each  project  subject  to  evaluation 

-  -  Other  managerial  and  technical  representatives  as  requested  by  the  government. 


c.  Each  organization  should  plan  to  provide  facilities  for  conducting  the  on-site  evaluation 


Figure  B-17:  U.S.  Army  CECOM  RFP  SCE  Text,  Section  L 
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Pii 


^coniinue 


IL-XX.X  SOFTWARE  PROCESS  RISK  EVALUATION.  K 

The  offeror  shall  provide  the  following  information  to  facilitate  an  evaluation  of  the  software  development;' 
process  using  the  SPRE  methodology:  p|| 

a.  The  offeror  shall  Identify  each  organization  that  will  be  involved  in  the  development  effort  described  ir^g 
the  proposal,  along  with  a  brief  description  of  the  effort  each  organization  will  perform.  For  the  purpose^^^^ 
of  this  and  all  the  following  bullets,  an  organization  is  defined  as  each  geographically  separate  corporation||p 
division,  or  site. 

b.  The  offeror  shall  provide  a  description  of  the  system  characteristics  as  it  applies  to  the  software  develop 
opment.  In  addition,  the  offeror  shall  identify  the  following  for  the  proposed  software  development  as  it  app^ 
plies  to  each  organization  involved  in  the  development  effort:  WA 

-  -  Size  of  software  effort  ^8 


-  Language(s)  to  be  used 
^8-  -  Tools/methodologies  to  be  used. 

^Sc.  The  offeror  shall  provide  an  overall  plan  for  the  development  of  software,  to  Include  software  developec 
^pby  all  organizations  on  the  offeror's  team.  This  plan  shall  include  a  discussion  of  newly  developed  soft- 
^pware.  Commercial  Off  The  Shelf  (COTS)  software,  Non-Development  Item  (NDI)  software,  and  the  inter 
^pfaces  between  them.  ^ 

^pd.  The  offeror  shall  provide  a  description  of  the  software  development  process  to  be  employed  for  the  de  ^ 
Wvelopment  effort  described  in  the  proposal.  This  description  shall  include  discussion  of  all  the  organize  » 
options  involved  In  the  development  effort.  If  the  described  process  is  different  from  processes  currently  ir » 
Mplace  (i.e.,  process  being  used  by  each  organization  on  the  offeror's  team),  the  offeror  shall  include  a  plar^J 
^pfor  implementing  the  described  process.  WM 


e.  Each  organization  identified  above  shall  provide  an  organization  chart  t  hat  describes  the  organizationa 
structure  from  upper  management  down  to  the  software  project  leader  level.  ||| 


If.  Each  organization  shall  provide  project  profiles  and  questionnaire  responses  on  six  software  develop^® 
|ment  projects  either  completed  during  the  past  year  or  currently  in  progress,  similar  in  scope  and  magni^B 
|tude  to  the  proposed  project,  and  from  the  site  and  organization  where  the  proposed  project  will  b^^: 
iperformed.  If  this  number  is  not  available,  consider  the  submission  of  projects  that  do  not  match  the  profiles 
las  closely.  The  format  to  be  used  for  completing  the  project  profiles  is  outlined  In  Attachment  II.  The  ques-^B 
pionnalre  responses  shall  represent  the  answers  to  the  questions  posed  in  Attachment  I.  The  answers  shal^p 
be  recorded  on  copies  of  the  form  provided.  W§i 


■Attachment  I  -  Software  Process  Maturity  Questionnaire 
wAttachment  II  -  Project  Profile  Form 


Figure  B-18:  U.S.Army  CECOM  RFP  SCE  Text,  SECTION  L  (cont’d) 
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|(2)  Specific  Criteria  -  Technical.  The  Technical  area  Is  divided  into  the  following  items  which  are  listed  ir[;TJ 
descending  order  of  importance. 

(x)  SOFTWARE  PROCESS  RISK  EVALUATION. 

The  Government  will  use  the  Software  Process  Risk  Evaluation  (SPRE)  to  evaluate  the  process  capal 
bility  of  each  offeror.The  SPRE  methodology  is  consistent  with  the  Software  Engineering  Institute’s  Soft| 
ware  Capability  Evaluation  methodology.  The  offeror’s  process  will  be  evaluated  against  the  Capability 
Maturity  Model  as  defined  in  CMU/SEI-93-TR-24  and  CMU/SEI-93-TR-25  to  determine  the  risks  associl 
ated  with  the  ability  of  that  process,  when  followed,  to  produce  quality  software  on  schedule  and  withirl 
budget.  The  target  process  capability  to  be  used  for  the  evaluation  consists  of  the  following  key  process 
areas: 

Software  Requirements  Management 
Software  Project  Planning 

Software  Project  Tracking  and  Oversight  Software  Subcontract  Management 
Software  Quality  Assurance 
Software  Configuration  Management 
Training  Program 
Software  Product  Engineering 
Peer  Reviews 


Figure  B-19:  U.S.Army  CECOM  RFP  Text,  Section  M 


B.3.1  Other  Examples 

The  examples  below  have  been  collected  from  various  agencies  and  pub¬ 
lications. 
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Jbu  Announcement 


22.  XXXXX  BBBBBB  CCCCCCCC  DDDDDD  (YYYYY) 

he  Government  Agency  has  limited  the  competition  for  the  XXXXX  BBBBBB  CCCCCCCC  DDDDDD 
(YYYYY).  Within  this  limited  competition,  vendors  will  be  prequalified  through  minimum  qualification  cri- 

eria  published  in  this  announcement.  The  intent  of  this  prequalification  process  is  to  assure  XYZ  Gov¬ 
ernment  Agency  are  truly  capable  of  successfully  performing  the  YYYYY  acquisition,  the  agency  has 
established  minimum  qualification  criteria  which  are  NOT  UNDULY  RESTRICTIVE.  However,  the  Gov¬ 
ernment  reserves  the  right  to  waive  minimum  qualifications  if  it  would  otherwise  enhance  competition. 
The  planned  date  for  the  award  of  YYYYY  contract  is  Month  19XX. 

Vendors  are  required  to  certify  in  writing  that  they  meet  the  stated  criteria.  As  a  means  of  assuring  fair¬ 
ness  to  potential  YYYYY  offerors  that  meet  the  minimum  qualification  criteria,  during  the  YYYYY  pre-con 
[tract  award  process,  any  offeror  found  not  to  have  met  the  minimum  qualification  criteria  will  be 
disqualified  from  continuing  in  the  YYYYY  competition.  The  exception  to  disqualification  will  be  the  Gov¬ 
ernments  waiver  of  the  minimum  qualifications  under  the  conditions  stated  above.  What  follows  are  the 
iGovernment’s  MINIMUM  QUALIFICATION  CRITERIA: 


1 .0  Operational  System 

Eligible  vendors  shall  have  deployed  a(n) _ 

ito . .  prior  to  the  due  date  of  this  CBD  notice. 


^system  which  is  operational  and  being  used 


I  a)  The  qualifying _ system  shall  perform _ ^functions . 

f  b)  The  qualifying  system  shall  process . 

c)  The  qualifying  system  shall  perform . 

d)  The  qualifying  system  shall  have  associated  system  documentation  to  include  training,  main¬ 
tenance  and  operator  manuals. 

Deliverables: 

§ 

b  substantiate  the  ability  to  meet  criteria  1 .0  above,  the  vendor  shall  supply  the  YYYYY  Contracting 
Officer  the  following  items: 

•  Customers  references  for  the  qualified _ system(s)  which  includes  a  customer  point 

of  contact,  customer  address,  customer  telephone  number,  and  the  associated  contract  name  and  num¬ 
ber. 

•  The  vendor  shall  provide  a  list  of  the  qualifying  system{s)  deliverable  documentation  as  well  as 
a  copy  of  the  operator  and  maintenance  manuals  for  that  system(s), 

NOTE'  The  Government  reserves  the  right  to  contact  the  user/procurer  of  the  qualifying  system(s)  to  ver 
ify  compliance  with  the  minimum  qualification  criteria.  The  Government  also  reserves  the  right  to  request 
further  system  documentation  from  the  vendor. 


Figure  B-20:  Other  Examples:  CBD  Announcement  SCE 
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V 


_  raining: 

The  •  system  meeting  criteria  1 .0  above  shall  currently  provide  the  capability  to 
.perform _ training . 


SDeliverables: 

To  substantiate  the  ability  to  meet  criteria  1.1  above,  the  vendor  shall  supply  to  the  YYYY  Con¬ 
tracting  Officer  the  following  Items: 


•  Commercially  available  literature  describing  the . 
of  the  qualifying  system 


Jralning  and. 


1.2  Second  Sub-item: 

The _ system  meeting  criteria  1 .0  above  shall  currently _ 

Deliverables: 

To  substantiate  the  ability  to  meet  criteria  1.2  above,  the  vendor  shall  supply  the  YYYYY  Con- 
ractlng  Officer  the  following  items: 

•  _ diagram,  of  the  system  meeting  criteria  1 .0  above, . 

•  A  copy  of  the _ 


M2.0  Software  Capability  Evaluation  (SCE)  Experience: 

^Background-  The  contractor  for  the  YYYYY  shall  perform  software  engineering  activities  including,  but 
inot  limited  to,  requirements  analysis,  design,  implementation,  integration,  test,  distribution.  Installation, 
^enhancement,  correction,  upgrade,  and  maintenance  of  software.  The  YYYYY  contractor  shall  be  both 
^knowledgeable  and  capable  in  software  engineering  skills  and  practices  to  perform  these  activities.  The 
^Government  Agency  intends  to  evaluate  the  software  engineering  capabilities  of  offerors  as  part  of  its 
Bsource  selection  process  for  YYYYY,  and  plans  to  use  the  results  In  its  determination  of  award. 


^pTo  ensure  that  offerors  are  cognizant  of  practices  for  evaluating  software  capabilities,  offerors  shall  sat- 
■isfy  the  following  requirement  to  participate  in  source  selection  activities  for  YYYYY: 

n  independent  SCE  shall  be  performed  prior  to  Government  Agency  conduct  of  a  software  capability  1 
evaluation  related  to  the  YYYYY  acquisition.  The  SCE  shall  be  in  accordance  with  the  conditions  out-  H 
lined  in  sections  a,  b,  c  and  d  below.  I 

a)  The  onsite  portion  of  the  SCE  was/shall  be  conducted  after  Month,  Day,  Year,  but  no  later  I 

Ihan  XX  calendar  days  after  the  issuance  of  the  final  version  of  the  YYYYY  Request  for  Proposal.  I 

b)  The  SCE  was/shall  be  performed  using  The  Capability  Maturity  Model  for  Software,  Version  I 
1 .1  as  described  in  CMU/SEI-93-TR-24  and  Key  Practices  of  the  Capability  Maturity  Model  Version  1.1  I 
mas  described  in  CMU?SEI-93-TR-25.  Both  documents  were  issued  by  the  Software  Engineering  Institute! 
■(SEI)  of  Carnegie  Mellon  University  in  February  1 993.  1 


Figure  B-21:  Other  Examples:  CBD  Announcement  SCE  (cont’d) 
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rom  a  request  hor  h^roposai 


Section  C 


''A/AfyAk  '< 


3.1.X.X1 


The  Contractor  shall  be  experienced  in  software  engineering  and  have  and  ongoing  program  for  improvinc 
their  software  process  in  accordance  with  the  Software  Engineering  Institute’s  (SEI)  Capability  Maturit; 
Model  (CMM).  The  contractor  shall  provide  a  Software  Process  Improvement  Plan  (SPIP)  for  improvinc 
their  software  process  through  the  contract  life.  After  contract  award,  the  Government  (or  Government  Rep 
resentative)  reserves  the  right  to  conduct  additional  Software  Capability  Evaluations,  and  request  statue 
reports  of  the  software  process  improvement  activities. 


''  'y'  VW  // 


Figure  B-22:  Other  Examples:  CBD  Announcement  SCE  (cont’d) 


I  ne  vendor  snail  also  suomit  a _ description  o  _ 

|no  later  than  <x>  days  following  submission  of  the  offeror’s  certification  letter. 

In  their  decision  to  take  part  in  the  YYYY  acquisition,  potential  offerors  are  advised  that  the  governmeni 
plans  to  require  industry  acceptance  of  the  following  condition: 

jRecognizIng  that  offerors  have  a  statutory  right  to  challenge  government  procurement  decisions  under . 

dditional  information  concerning  this  planned  acquisition  will  be  provided  at  the  soonest  possible  date  via 
jseparate  announcement.  / 


frhis  is  not  a  Request  for  Proposal  synopsis  nor  is  the  government  agency  seeking  or  accepting  unsolicitec 
^proposals. 

^  he  required  certification  for  the  YYYY  minimum  qualification  criteria  must  be  forwarded  to  Mr.  <contractin 
'  fficer>  no  later  than  <month>,  <day>,  <yeai>  at: 

U.S.  mall  address> 
fax  number> 


Figure  B-23:  Other  Examples:  CBD  Announcement  SCE  (cont’d) 
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rom  a  Hequest  hor  Proposal  •  section 


.X.X  Annex  D  —  SEI  Certification  /  Currency 


ny  software  developed  and  delivered  shall  be  produced  by  SEI  Level  3  certified  contractor.  The  offero 
>hall  provide  proof  of  current  level  3  certification,  at  the  time  of  submission,  for  all  software  contractors  or 
his  contract. 


Figure  B-24:  Other  Examples:  RFP,  Section  L 
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Appendix  C  Current  DoD  Policy  Directives:  SCE 

This  appendix  contains  the  text  of  existing  DoD  policy  documents  obtained  from  the  USA, 
USN,  and  the  USAF.  These  documents  are  provided  in  their  existing  formats  (e.g.,  USA  and 
USN  doucuments  are  draft).  Questions  regarding  these  documents  should  be  addressed  to 
the  referenced  organizations  and  points  of  contacts. 
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(Draft)  CECOM-R  11 XX 
Headquarters 

U.S.  Army  Communications-Electronics  Command 
Fort  Monmouth,  New  Jersey  07703-5000 


CECOM  Regulation  1  Dec  1991 

No.  11-XX 

Army  Programs 

Software  Capability  Evaluation 

Issue  of  changes  to  this  regulation  by  other  Command  elements  is  prohibited  unless  specifi¬ 
cally  approved  by  Commander,  CECOM. 

ATTN:  AMSEL-RD-SE-R-CRM 


Paragraph 


Purpose  1 

Scope  2 

References  3 

Terms  4 

Applicability  5 

Policy  6 

Responsibilities  7 


1 .  Purpose.  This  regulation  establishes  a  CECOM  Software  Capability  Evaluation  (SCE)  pol¬ 
icy  for  software  procured  for  Mission  Critical  Defense  Systems  (MCDSs). 

2.  Scope.  This  regulation  applies  to  all  CECOM  managed  MODS.  It  applies  to  software  pro¬ 
curements  whether  procured  separately  or  in  conjunction  with  other  items  or  services.  It 
applies  to  mission  critical  software  procurements  for  development  or  maintenance  con¬ 
tracts. 
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3.  References.  Required  publications  are  as  follows: 

a.  CMU  (Carnegie-Mellon  University)/SEI  (Software  Engineering  lnstitute)-87-TR- 
23  (ESD-TR-87-186),  A  Method  for  Assessing  the  Software  Engineering 
Capability  of  Contractors,  Preliminary  Version,  September  1987.  (Available 
from  the  Defense  Technical  Information  Center  as  Report  ADA1 87230). 

b.  CMU/SEI-90-TR-XX,  Software  Capability  Evaluation  Implementation 
Handbook:  Source  Selection,  Draft  with  Army  Appendix  B. 

c.  DoD  Instruction  5000,2,  Defense  Acquisition  Program  Procedures  23  Feb  91 . 

d.  Draft  AMC  Army  Implementation  Instructions:  Use  of  Software  Capability 
Evaluation  in  Source  Selection  16  Aug  1991 . 

e.  CECOM-R  11-  XXX,  Software  Acquisition  and  Support  for  Mission  Critical 
Defense  Systems  (Draft). 

f.  Communications-Electronics  Command  Management  Plan  for  Using  the 
Software  Capability  Evaluation  November  1 991 . 

4.  Terms.  For  the  purpose  of  this  regulation,  the  following  terms  shall  apply: 

a.  Mission  Critical  Defense  System  (MCDS):  An  Army  system  involving 
Intelligence/Electronic  Warfare,  Command  and  Control,  Communication,  Fire 
Support,  Maneuver  Control,  or  other  tactical  weapon  systems  managed  by  or 
supported  by  CECOM,  including  its  life-cycle  support  environment,  that  is  a 
Theater/Tactical  and/or  Strategic  resource  and  is  managed  in  accordance  with 
AR-70  series  regulations.  (Definition  derived  from  the  Under  Secretary  of 
Defense  memorandum,  subject:  Acquisition  of  Computer  Resources,  dated  4 
Mar  83.) 

b.  Mission  critical  Software:  1)  A  set  of  computer  programs,  code,  procedures  and 
associated  documentation  concerned  with  the  operation,  maintenance  and 
support  of  a  MCDS's  computer  system.  (Definition  derived  from  JCS  Pub  1-2, 1 
Dec  89);  2)  A  combination  of  associated  computer  instructions  and  computer 
data  (object)  definitions  required  to  enable  the  computer  hardware  to  perform 
computational  or  control  functions  (for  the  MCDS)  (Definition  derived  from 
DOD-STD-2167A). 

c.  Source  Lines  of  Code  (SLOC):  Code  line  is  language  independent  (assembly 
or  HOL)  with  declarative  and  executable  statements  included  in  the  count, 
exclusive  of  comments,  and  measured  at  one  statement  per  line  of  code. 
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'  5.  Applicability:  The  applicability  for  mandatory  inclusion  of  SCE  requirements  in  a  Request 

For  Proposal  (RFP)  are  whenever  any  four  of  the  following  seven  conditions  are  met: 

a.  Solicitation  for  proposal  on  an  Engineering-Development  (ED)  or  Full-Scale 
Development  (FSD)  contract.  Contracts  for  a  DemonstrationA/alidation  phase 
are  also  subject  to  an  SCE  if  any  portion  of  the  software  if  intended  for  reuse 
during  a  follow-on  ED  or  FSD  contract  phase.  This  includes  contracts  for 
Engineering  Change  Proposal  (ECP). 

b.  Any  portion  of  the  software  is  subcontracted.  The  strong  likelihood  of  software 
contracting,  bases  on  knowledge  of  bidder  prior  to  receipt  of  proposals,  is 
sufficient  to  meet  this  criteria. 

c.  Terms  of  contract  include  mission-critical  software  components. 

d.  Size  of  software  to  be  developed,  including  non-ope rational  (support  and  test) 
software  when  specified  for  delivery  as  a  Computer  Software  Configuration 
Item  (CSCI),  is  at  least  50,000  Source  Lines  Of  Code  (SLOC),  or  when 
delivering  integration  software  for  Non-Developmental  Items  (NDI)/Commercial 
Off-The-Shelf  (COTS)  software  where  the  NDI/COTS  software  exceeds  on  third 
of  the  software  to  be  delivered. 

e.  Total  contract  cost  exceeds  $1 0,000,000. 

f.  The  contract  duration  is  specified  as  greater  than  two  years. 

g.  The  software  development  schedule  is  a  critical  risk  factor  in  the  contract. 

6.  Policy.  ON  all  applicable  procurements: 

a.  An  evaluation  of  an  offeror's  software  development  process  capability  shall  be 
performed  as  part  of  the  source  selection  process.  Software  development 
process  capability  will  be  determined  through  an  evaluation  of  the  offeror's 
software  development  process  maturity  and  extent  of  software  process  risk  as 
measured  by  certification  of  his  responses  to  the  questionnaire  in  CMU/SEI-87- 
TR-23  as  verified  by  on-site  validation.  This  evaluation  shall  be  referred  to  as 
“Software  Engineering  Capability  Evaluation  (SCE)”. 

b.  The  evaluation  will  result  in  a  report  indicating  the  offeror's  strengths  and 
weaknesses  in  the  Key  Process  Areas.  The  use  of  this  report  in  the  source 
selection  process  will  be  documented  in  the  approved  Source  Selection  Plan 
and  in  the  RFP.  The  usage  of  the  SCE  is  to  be  as  a  factor  in  the  Source 
Selection  Evaluation  Board  (SSEB).  Additional  guidance  on  preparation  and 
use  of  the  SCE  is  available  in  the  Army  Implementation  Instructions. 

c.  Waivers:  All  procurements  involving  the  acquisition  of  MCDS  software  require 
the  us  of  an  SCE  or  an  approved  waiver. 

(1)  Requests  for  Waiver  will  be  evaluated  on  the  basis  of  the  relative 
importance  of  the  software  effort  or  overall  cost-benefit. 
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(2)  Requests  for  Waiver  will  be  prepared  by  the  Requiring  Activity, 
reviewed  by  the  Director,  CEDOM  Software  Engineering  (SED), 
ATTN:  AMSEL-RD-SE-D,  prior  to  the  release  of  the  solicitation.  If  the 
waiver  is  not  approved  by  the  Director  CECOM  SED,  the  disapproval 
must  be  reviewed  and  signed  by  the  CG,  CECOM. 

(3)  Waiver  requests  should  include,  as  a  minimum,  a  discussion  of 
each  of  the  conditions  under  paragraph  5,  “Applicability.” 

d.  Requests  for  exemptions  from  an  SCE  received  from  individual  offerors  will  be 
reviewed  by  the  Requiring  Activity  and  a  determination  made  by  the  Director, 
CECOM  SED  as  to  its  applicability  to  the  current  solicitation. 

7.  Responsibilities: 

a.  The  Chief,  Requiring  Activity  will: 

(1)  Implement  the  provisions  of  the  regulation  for  all  applicable 
systems,  including  funding  for  SCEs. 

(2)  Include  the  SCE  requirement  in  acquisition  plans,  acquisition 
strategies,  and  solicitation  documents  (including  evaluation  plans)  for 
the  applicable  procurements. 

(3)  Obtain  a  waiver  of  SCE  application,  if  necessary,  prior  to  issuance 
of  a  solicitation. 

b.  The  Directorate  for  procurement  will  review  all  applicable  software  procurement 
data  packages  for  a  requirement  for  an  SCE.  Solicitations  will  not  be  issued 
without  and  SCE  requirement  or  ^n  approved  waiver. 

c.  The  Director,  Concurrent  Engineering  Directorate  (CED)  will: 

(1)  Provide  qualified,  trained  Software  Test  and  Software  Quality 
Assurance  personnel  for  inclusion  as  CET  members. 

d.  The  Chiefs,  Source  Selection  Authorities  will  ensure  that  applicable 
requirements  of  this  regulation  are  included  in  the  Source  Selection  Plan. 

e.  The  Director,  Software  Engineering  (SED)  will: 

(1)  Review  procurement  data  packages  to  determine  whether  SCE  is 
applicable. 

(2)  Appoint  a  Capability  Evaluation  Team  (CET)  Leader  for  each 
project  requiring  an  SCE. 

(3)  Approve  the  membership  of  the  CET, 

(4)  Maintain  an  SCE  competence,  provide  guidance,  and  serve  as  the 
SCE  focal  point  within  CECOM. 
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(5)  Review  and  approve  Requests  for  Waiver  or  recommend 
disapproval  to  the  CG,  CECOM, 

(6)  Maintain  a  centralized  repository  of  SCE  data. 

f.  The  CET  Leader  will: 

(1 )  Assist  the  Requiring  Activity  in  their  implementation  and  support  of 
SCE. 

(2)  Coordinate  and  recommend  CET  members  from  CECOM, 
PEO/PM,  and  other  sources  as  appropriate  to  the  Director,  SED  for 
approval. 

(3)  Arrange  CET  Team  visits  and  prepare  CET  Team  reports. 

g.  The  Chief  of  a  non-CECOM  organization  such  as  a  Program  Executive  Officer 
(PEO)  or  non-CECOM  Program/Project  Manager  (PM),  will  comply  with  the 
requirements  of  subparagraph  6.a  above  in  order  to  have  systems  accepted  for 
software  maintenance  and  support  by  CECOM. 


The  proponent  of  this  publication  is  the  U.S.  Army 

Communications-Electronics  Command.  Users  are  invited  to  send  comments  on  DA  Form 
2028  (Recommended  Changes  to  Publications  and  Blank  Forms)  to  Commander,  CECOM, 
ATTN:  AMSEL-RD-SE-R-CRM,  Fort  Monmouth,  NJ  07703-5207. 


Official:  ALFRED  J.  MALLETTE 
Major  General,  USA 
Commanding 


AUBREY  D.  CRAIG 
Colonel,  GS 
Chief  of  Staff 


Distribution: 

To  be  distributed  in  accordance  with  SEL  Form  1130  requirements  for 
CECOM  Regulations,  Army  Programs,  plus 

AMSEL-RD-SE-R-CRM . 12  Record  Set  File,  ATTN: 

SELFM-RM-ER.6 
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Department  of  the  Navy 

Naval  Information  Systems  Management  Center 
1225  Jefferson  Davis  Highway 

Ariington,  Virginia  22202 


5230 

SER:  03/95/0072 
Jan  24, 1995 


From:  Commander,  Naval  Information  Systems  Management  Center 
Subj:  DRAFT  SOFTWARE  PROCESS  IMPROVEMENT  POLICY 
Enel:  (1)  Draft  SECNAVINST  5234,  Software  Process  Improvement  Policy 

1 .  I  request  your  support  in  reviewing  the  enclosed  initial  draft  Software  Process  Improve¬ 
ment  Policy.  I  would  like  to  recognize  Mr.  Jim  Stine  of  FMSO  and  his  staff  for  the  outstand¬ 
ing  technical  assistance  which  they  provided  in  preparation  of  this  document. 

2.  Please  ensure  widest  distribution  of  this  initial  draft  within  your  organization  for  review  and 
comment.  Please  provide  comments  by  3  March  1995. 

3.  If  you  have  questions,  please  feel  free  to  call  me.  Ms.  Margaret  Powell  is  my  point  of  con¬ 
tact  for  your  staff.  She  can  be  reached  via  e-mail  at  powellma.ntrprs@navair  navy.mil  or 
phone  at  (703)  602-6906  (DSN  332).  Comments  may  be  e-mailed  to  Ms.  Powell  or  faxed 
to  her  at  (703)  602-4722. 

I  J.  G.  Hekman 

Rear  Admiral,  SC,  USN 

Distribution: 

CNO  (N2,  N4,  N6B) 

COMNAVAIRSYSCOM 
COMNAVFACENGCOM 
COMS  PAWARSYSCOM 
COMNAVSEASYSCOM 
COMNAVCOMTELCOM  (Code  N624) 

PEG  (A) 

PEG  (SCS) 

PEG  (TAD) 

PED(T) 

PEG  (CU) 

PEG  (MIW) 

PEG  (SUB) 
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PEO  (USW) 

PEO  (JAST) 

DRPM(AAA) 

DRPM  (AEGIS) 

DRPM(SSP) 

NCCOSC 

NAVMEDINFOMGMTCEN  (CODE  03) 

NAWC 

SUPERS 

OCEANOGRAPHER 

CNR 

NSWC 

NUWC 

NRL  (CODE  5500) 

HQMC  (C4I) 

FMSO  (Code  9ED,  902) 

NRaD  SEPO  (Code  0202) 

NSWC  (Code  6002A2) 

NAWCCAD  (Code  4573) 

NAWC  (Code  4FOOOOD)) 

MCTSSA  (Q-01) 

DASN(C4l/EW/Space)ABM  (Attn  F.  Ford) 
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DRAFT 


SECNAVINST  5234 
NISMC 
13  January  1995 


SECNAV  Instruction  5234 

From:  Secretary  of  the  Navy 

Subj:  Department  of  the  Navy  (DON)  Software  Process  Improvement 

Policy 

Ref: 

a.  DODD  5000.1  of  23  Feb  91 ,  “Defense  Acquisition”  (NOTAL) 

b.  DODI  5000.2  Change  1  of  26  Feb  93,  “Defense  Acquisition  Management 
Policies  and  Procedures”  (NOTAL) 

c.  DoD  5000.2-M  of  23  Feb  91 ,  “Defense  Acquisition  Management  Documentation 
and  Reports”  (NOTAL) 

d.  SECNAVINST  5000.2A  of  9  Dec  92,  “Implementation  of  Defense  Acquisition 
Management  Policies,  Procedures,  Documentation,  and  Reports”  (NOTAL) 

e.  SECNAVINST  5200.32A  of  3  May  93,  “Acquisition  Management  Policies  and 
Procedures  for  Computer  Resources”  (NOTAL) 

f.  DODD  8120.1  of  14  Jan  93,  “Life-Cycle  Management  (LCM)  of  Automated 
Information  Systems  (AISs)”  (NOTAL) 

g.  DODI  8120.2  of  14  Jan  93,  “Automated  In  formation  System  (AIS) Life-Cycle 
Management  (LCM)  Process,  Review,  and  Milestone  Approval  Procedures” 
(NOTAL) 

h.  SECNAVINST  5231. 1C  of  10  Jul  92,  “Life  Cycle  Management  Policy  and 
Approval  Requirements  for  Information  System  Projects” 

Enel: 

1 .  Definitions 

2.  Implementation  Guidance 

1 .  Purpose.  To  establish  a  Department  of  the  Navy  (DON)  Software  Process  Improvement 

(SPI)  policy  and  implement  a  comprehensive  DON  Software  Process  Improvement 

(SPI)program  for  systems  managed  under  references  (a)  through  (h). 
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1.  Background 

a.  Software  has  become  a  major  part  of  all  systems  which  the  DON  develops  or 
acquires.  Traditionally,  software  requires  intensive  management  oversight  to 
ensure  a  quality  product.  Software  defects  are  a  major  cause  of  concern- 
-products  are  not  delivered  when  expected:  software  does  not  perform  as 
expected;  or  costs  are  significantiy  greater  than  expected.  The  DON 
recognizes  these  concerns  and  is  committed  to  minimizing  the  adverse  impacts 
associated  with  inefficient,  non-performant  software.-  Since  DON  obtains 
software  either  through  Government  Agencies  or  by  acquisition  via  a 
contractual  vehicle,  we  must  ensure  that  software  process  improvement  is 
addressed  within  both  organic  activities  and  contracted  organizations 

b.  The  Software  Engineering  Institute  (SEI)  has  developed  the  Capability  Maturity 
Model  (CMM)  which  forms  a  framework  for  software  process  improvement.The 
model  defines  five  levels  of  maturity,  each  level  building  on  successive 
foundations  for  increased  process  maturity.  Each  level  has  a  number  of  key 
process  areas  associated  with  it  (requirements  management,  software  project 
planning,  software  configuration  management,  software  quality  assurance, 
etc.)  and  can  be  used  to  evaluate  both  organic  and  non-organic  resources.  The 
SEI  is  currently  evolving  their  program  to  adopt  CMM-based  appraisal  (CBA) 
techniques.  The  Internal  Process  Improvement  (IPI),  adapted  from  the 
Software  Process  Assessment  (SPA)  methodology,  can  be  used  to  evaluate 
organic  capabilities.  Because  the  CBA  methodology  is  still  evolving,  the  IPI  is  in 
field  test,  and  other  appraisal  methodologies  may  not  yet  be  defined,  this 
instruction  will  use  the  terms  assessment  and  SPA  to  designate  an  internal 
appraisal  and  evaluation  and  SCE  to  designate  an  external  appraisal  of  an 
organization. 

c.  The  Software  Process  Assessment  (SPA)  methodology  is  used  to  evaluate 
organic  capabilities.  To  initiate  an  SPA,  a  software  development  activity  must 
address  risks  and  weaknesses  associated  with  its  software  development 
capabilities.  Once  risks  and  weaknesses  have  been  identified,  a  corrective  plan 
of  action  is  developed  and  implemented.  The  SPA  approach  encourages 
objective,  open,  and  thorough  analysis  because  results  are  for  internal  use  only 
and  may  be  published  only  as  part  of  an  aggregate  reporting  requirement. 

d.  The  Software  Capability  Evaluation  (SCE)  process  defines  a  method  to 
evaluate  a  developer's  software  process.  Based  on  identified  strengths  and 
weaknesses,  the  evaluation  allows  the  acquirer  to  determine  the  risks 
associated  with  an  organization's  software  development  environment  and  the 
potential  for  repeatedly  developing  quality  software.  The  SCE  can  be  used 
during  the  solicitation  phase  to  determine  risks,  or  post-award  to  determine 
award  fee  schedules.  It  is  envisioned  that  this  process  would  become 
institutionalized  so  that  it  could  be  applied  to  all  sources  of  government  software 
acquisition,  either  organic  or  contractor. 

2.  Applicability  and  Scope.  This  policy  applies  to  all  DON  organizations  that  acquire,  devel¬ 
op,  or  maintain  software  or  software  components. 

3.  Definitions.  Definitions  of  terms  used  in  this  instruction  are  contained  in  enclosure  (1). 
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4,  Policy.  The  following  policies  govern  the  DON  SPI  Program: 

a.  Software  Process  Appraisal.  All  DON  activities  with  twenty  or  more  personnel 
directly  involved  in  the  development  or  maintenance  of  software  or  software 
components  (subsequently  referred  to  as  the  “software  Organization”),  or  an 
annual  software  development  and/or  maintenance  budget  of  at  least  two  million 
dollars  shall  conduct  an  appraisal  of  their  software  processes.  For  consistency, 
this  appraisal  shall  be  based  upon  the  Software  Engineering  Institute  (SEI) 
Capability  Maturity  Model  (CMM).  Guidance  for  this  appraisal  is  contained  in 
enclosure  (2). 

(1)  Initial  CMM  based  appraisals  shall  be  completed  prior  to  1 
September  1996.  Follow-on  appraisals  shall  De  performed  every 
three  years. 

(2)  Exemption.  A  software  organization  scheduled  closure  or  planned 
to  be  significantly  reduced  within  two  years  from  the  effective  date  of 
this  instruction,  such  that  on  1  January  1997,  it  would  be  under  the 
minimum  criteria  established  in  paragraph  5a,  is  exempt  from  this 
policy. 

b.  Software  Process  Improvement  Program.  All  software  organizations  meeting 
the  criteria  of  paragraph  5a  shall  define,  develop  and  implement  a  software 
process  improvement  program  which  uses  a  continuous  improvement  cycle. 
The  improvement  process  shall  start  with  the  organizational  appraisal.  Each 
software  organization  shall  develop  and  implement  a  software  process 
improvement  plan  (SPI  Plan)  based  on  the  findings  of  the  initial  appraisal. 
Additional  guidance  is  included  in  enclosure  (2). 

c.  Software  Capability  Evaluations  (SCE).  All  DON  organizations  that  contract  for 
the  design,  development  or  mainteniance  of  software  or  software  components 
shall  conduct  SCEs.  The  SCE  may  be  used  to  conduct  a  risk  evaluation  to 
establish  a  baseline  of  the  bidding  contractor's  software  practice,  and/or  as  a 
post-award  evaluation  to  determine  fee  award  schedules.  For  consistency,  the 
evaluations  shall  be  based  upon  the  Software  Engineering  Institute  (SEI) 
Capability  Maturity  Model  (CMM).  Implementation  Guidance  is  provided  in 
enclosure  (2), 

(1)  The  SCE  methodology  shall  be  an  integral  part  of  the  source  selection 
process  for  any  procurement  that  meets  both  the  cost  criteria  defined  in 
paragraph  2a,  and  any  one  of  the  other  six  criteria-  defined  in  paragraph  2b. 

(2)  SCE  Criteria 

(a)  Cost.  The  total  value  of  the  contract,  including  all  options,  as 
estimated  to  exceed  ten  million  dollars. 

(b)  Other  criteria.  Various  factors  which  may  be  considered  high 
risk  areas  for  high  cost  systems  must  be  considered  when 
determining  the  need  for  an  SCE.  In  addition  to  cost,  at  least  one  of 
the  following  criteria  must  be  met: 
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(1)  Criticality.  The  RFP  includes  mission  critical  Software. 

(2)  Size.  The  size  of  the  software  to  be  delivered  is  at  least  200, 000 
Source  Lines  of  Code  (SLOC). 

(3)  Duration.  The  contract  duration  is  specified  as  greater  than  two 
years. 

(4)  New  venture.  A  major  component  of  the  total  system,  including 
its  software  functionaiity,  is  considered  to  be  unprecedented. 

(5)  Critical  software  development  schedule.  The  software 
development  schedule  is  a  critical  item. 

(6)  Subcontractors.  It  is  anticipated  that  more  than  one-half  of  the 
software  is  to  be  subcontracted.  The  strong  likelihood  of  software 
subcontracting,  based  upon  knowledge  or  prospective  offerors  prior 
to  receipt  or  proposals,  is  sufficient  to  meet  this  criteria. 

d.  Funding.  Effective  implementation  of  SPI  is  expected  to  accrue  savings  in 
software  costs.  As  software  organizations  improve  their  software  development 
capability  and  become  more  in  demand,  it  is  expected  that  SPI  program  costs 
will  be  absorbed  as  a  mission  funded  cost  of  doing  business.  Centers  of 
Excellence  will  be  reimbursed  on  a  fee-for-service  basis. 

e.  Software  management  indicators  and  metrics.  To  ensure  that  software 
process  improvement  is  achieved,  a  core  set  of  metrics  (size,  effort,  schedule, 
and  quality)  shall  be  implemented  to  measure  software  practices  and  products. 
The  core  set  of  metrics  may  be  extended  to  meet  specific  requirements. 


5.  Responsibilities.  All  DON  organizations  shall  ensure  compliance  with  this  instruction 
within  their  respective  programs  and  projects:  fund  and  staff  implementation  of  SPI  efforts 
at  their  SPI  activities:  and  identify  to  NISMC  the  names  and  size  of  all  qualifying  software 
organizations  within  60  days  of  the  effective  date  of  this  policy.  In  addition: 

a.  The  Assistant  Secretary  of  the  Navy  Research.  Development  and  Acquisition 
(ASN  (RD  &  A))  shall  ensure  the  SPI  policy  in  this  instruction  is  implemented. 

b.  The  Commander,  Naval  Information  Systems  Management  Center 
(COMNISMC),  designated  by  reference  (d)  as  the  DON  Software  Executive 
Official  (SEO),  shall: 

(1)  Be  responsible  for  implementation  and  maintenance  of  the  DON 
SPI  policy. 

(2)  Establish  and  chair  the  DON  Software  Engineering  Process 
Working  Group  (SEPWG)  as  a  subcommittee  of  the  DON  Software 
Executive  Officiais  Councii  (SEOC). 


CMU/SEI-95-TR-012 


©  1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


163 


Current  DoD  Policy  Directives:  SCE 


April  1996 


(3)  Designate  the  DON  representative  to  the  DISA  Center  for 
Software's  Software  Process  Improvement  Advisory  Group  (SPIAG). 

(4)  Confirm  Navy  and  Marine  Corps  nominations  of  Centers  or 
Excellence  for  DON  SPI  policy  support  and  assistance. 

(5)  Be  responsible  for  implementation  and  maintenance  of  the  DON 
Software  Metrics  policy. 

c.  The  Chief  of  Naval  Operations  (CNO)  shall: 

(1)  Properly  resource,  both  with  dollars  and  trained  personnel,  the 
SPI  activities  and  organizations  defined  in  this  instruction. 

(2)  Prioritize  software  process  improvement  funding  so  that  a 
continuous  improvement  discipline  can  be  implemented  and 
sustained. 

d.  The  Commandant  of  the  Marine  Corps  (CMC)  shall: 

(1)  Nominate  DON  SPI  Centers  of  Excellence  to  NISMC  for  review 
and  validation. 

(2)  Fund  and  staff  Marine  Corps  SPI  Centers  of  Excellence  and 
ensure  personnel  are  fully  qualified  in  specific  procedures  and  policy. 

e.  The  DON  SEPWG  shaW; 

(1)  Function  as  a  DON  working  group  to  develop  common  processes, 
procedures  and  documentation  for  use  in  implementing  this  policy. 

(2)  Include  membership  consisting  of  representatives  from  each  DON 
Software  Process  Improvement  (SPI)  Center  of  Excellence  and 
others  as  required. 

f.  Each  DON  SPI  CENTER  OF  EXCELLENCE  shall  consist  of  experienced 
personnel  who  are  trained  and  skilled  in  all  software  process  improvement 
areas  addressed  by  this  policy.  They  shall  be  the  primary  source  of  SCE,  SPA 
and  SPI  guidance,  trained  appraisers  and  evaluators  within  their  designated 
functional  domains.  They  shall  use  the  standard  processes  and  procedures 
developed  by  the  DON  SEPWG. 


g.  The  Commanders,  Naval  Systems  Commands  (SYSCOMs)  shall: 

(1)  Nominate  DON  SPI  centers  or  Excellence  to  NISMC  for  review 
and  validation. 
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(2)  Fund  and  staff  their  DON  SPI  Centers  of  Excellence  and  ensure 
personnel  are  fully  qualified  in  specific  procedures  and  policy. 

h.  Program  Executive  Officers  (PEOs),  Direct  Reporting  Program  Managers 
(DRPMs)  and  Program  Managers  (PMs)  shall: 

(1)  Integrate  this  SPI  policy  into  their  organizational  responsibilities. 

(2)  Incorporate  planning  for  SCEs,  SPA  and  SPI  into  their  acquisition 
strategy  and  program  plans. 

(3)  Contact  primary  DON  SPI  Centers  of  Excellence  for  guidance 
when  SCE,  SPA  or  SPI  criteria  are  met  and  coordinate  with  them  to 
schedule  and  identify  the  dollar  and  personnel  resources  necessary 
to  implement  this  SPI  Policy. 

(4)  Utilize  their  primary  DON  SPI  Center  of  Excellence  for  SCE,  SPA 
and  SPI  support  on  a  fee-for-service  basis. 


signature 


Distribution: 


stocking  info.  etc. 
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Capability  Maturity  Model®*^  (CMM®“):  The  Software  Engineering  Institute  (SEI)  developed 
a  structured  method  for  describing  the  maturity  of  a  software  organization.  It  is  a  description 
of  the  stages  through  which  software  organizations  evolve  as  they  define,  implement,  mea¬ 
sure,  control  and  improve  their  software  processes.  This  model  provides  a  guide  for  selecting 
process  improvement  strategies  by  facilitating  the  determination  of  current  process  capabili¬ 
ties  and  the  identification  of  the  issues  most  critical  to  software  quality  and  process  improve¬ 
ment.  It  is  documented  in  CMU/SEI-93-TR-24  “Capability  Maturity  Model  for  Software”  and 
CMU/SEI-TR-25  “Key  Practices  of  the  Capability  Maturity  Model". 

Contractor:  An  individual  partnership,  corporation,  or  association  that  contracts  with  another 
organization  to  design,  develop  and/or  manufacture  one  or  more  products.  This  includes  com¬ 
mercial  as  well  as  government  organizations. 

Critical  Software  Development  Schedule:  When  completion  of  software  is  on  the  critical 
path  of  a  project's  schedule,  the  project  is  deemed  to  have  a  critical  software  development 
schedule.  When  such  is  the  case,  even  minor  slips  to  the  software  development  schedule 
may  impact  system  integration,  testing,  and  final  delivery  of  the  product. 

Mission  Critical  Software:  That  software  which  is  developed  to  satisfy  system  requirements 
which  are  deemed  critical  by  the  contract  or  by  system  specifications.  Mission  critical  soft¬ 
ware  may  address  safety-critical,  security-critical,  or  privacy-critical  issues.  The  developer 
should  develop  a  strategy  to  assure  that  the  requirements,  design,  implementation,  and  oper¬ 
ating  procedures  for  the  identified  software  minimize  or  eliminate  the  potential  for  violations  of 
critical  requirements;  record  the  strategy  in  the  software  development  plan;  implement  the 
strategy;  and  produce  evidence,  as  part  of  required  software  products,  that  the  assurance 
strategy  has  been  carried  out. 

Software  Process  Improvement  and  Capability  dEtermination  (SPICE):  Spice  is  a  suite  of 
standards  on  software  process  assessment  being  developed  by  the  International  Standards 
Organization  (ISO).  In  addition  to  software  development  and  maintenance  practices,  the  stan¬ 
dard  will  also  be  concerned  with  people,  technology,  management  practices,  customer  sup¬ 
port,  and  quality.  The  SPICE  standard  will  consist  of  the  following  products:  Introductory 
Guide;  Baseline  Practices  Guide;  Assessment  instrument;  Process  Assessment  Guide;  Pro¬ 
cess  Improvement  Guide;  Process  Capability  Determination  Guide;  and  Assessor  Training 
and  Qualification  Guide.  The  suite  of  proposed  SPICE  standards  are  targeted  for  publication 
as  Technical  Reports  during  1 994.  A  pilot  testing  period  will  follow.  The  proposed  standards 
will  be  revised  and  submitted  for  consideration  as  an  ISO  standard  during  the  1 997  time  frame. 
The  SEI  is  actively  participating  in  the  SPICE  standardization  effort. 


Capability  Maturity  Modei  and  CMM  are  service  marks  of  Carnegie  Mellon  University. 
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DON  Software  Executive  Official  (SEO):  The  Department  of  Defense,  in  DoD  Instruction 
5000.2,  Part  6  Section  D,  Computer  Resources,  requires  each  DoD  Component  Acquisition 
Executive  to  designate  a  senior  level  Software  Executive  Official  (SEO)  who  will  monitor,  sup¬ 
port,  and  be  the  focal  point  for  Ada  usage  and  sound  software  engineering,  development,  and 
life-cycle  support  policy  and  practice.  SECNAVINST  5000.2A  designates  the  Commander, 
Naval  Information  Systems  Management  Center  (NISMC)  the  DON  SEO. 

DON  Software  Executive  Officials  Council  (SEOC):  Chaired  by  DON  SEO.  The  SEOC  con¬ 
sists  of  DON  Flag  /SES  level  representatives.  Their  purpose  is  to  advise  DON  SEO  on  soft¬ 
ware  management  and  technology  actions  with  the  DON. 

DON  Software  Engineering  Process  Working  Group  (DON  SEPWG);  A  subcommittee  of 
the  DON  SEOC.  A  technical  working  group  of  software  professionals  from  each  DON  Software 
Process  Improvement  Center  of  Excellence.  They  are  responsible  for  formulating  Navy  SPI 
policies,  processes,  and  procedures  to  ensure  standard  Implementation  of  this  policy  through¬ 
out  DON. 

DON  SPI  Centers  of  Excellence;  A  group  of  SPI  experts  that  have  responsibility  for  coordi¬ 
nating,  planning,  managing,  implementing  and  maintaining  these  SPI  policies  as  appropriate 
for  all  organizations  within  their  sphere  of  influence.  They  will  be  the  primary  source  for  assis¬ 
tance  in  defining  and  performing  the  software  capability  evaluations  and  software  process  ap¬ 
praisals. 

Software:  Computer  programs  and  computer  databases.  NOTE:  Although  some  definitions 
include  documentation,  this  instructioiVli^mits  the  definition  to  computer  programs  and  comput¬ 
er  databases  in  accordance  with  Defense  Federal  Acquisition  Regulation  Supplement 
227.401.  i 

Software  Engineering  Institute  (SEI);  The  Software  Engineering  Institute  (SEi)  is  a  federally 
funded  research  and  development  center  (FFRDC)  sponsored  by  the  Department  of  Defense 
through  the  Advanced  Research  Projects  Agency  (ARPA).  The  SEI  contract  was  competi¬ 
tively  awarded  to  Carnegie  Mellon  University  (CMU)  in  December  1984.  It  is  staffed  by  ap¬ 
proximately  200  technical  and  support  people  from  industry,  academia,  and  government.  The 
SEI  was  established  by  DoD  because  software  has  become  an  increasingly  critical  compo¬ 
nent  of  U.S.  defense  systems  and  because  the  demand  for  quality  software  produced  on  a 
schedule  and  with  budget  exceeds  its  supply.  The  SEI's  mission  is  to  provide  leadership  in 
advancing  the  state  of  the  practice  in  software  engineering  in  order  to  improve  the  quality  of 
systems  that  depend  on  software. 

Software  Capability  Evaluation  (SCE):  An  appraisal  by  a  trained  team  of  professionals  to 
identify  organizations  who  are  qualified  to  per  the  software  work  or  to  monitor  the  state  of  the 
software  process  used  on  an  existing  software  effort.  The  CE  is  the  structured  methodology 
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SEI  developed  to  conduct  an  evaluation  of  an  offeror’s  software  process  capability.  It  is  based 
upon  the  CMM.  It  measures  a  target  organization  against  the  Key  Process  Areas  (KPAs)  of 
the  CMM.  Analysis  of  the  KPAs  determines  the  maturity  of  an  organization's  software  engi¬ 
neering  process  as  an  indicator  of  capability  or  risk  involved  if  selected  to  deliver  a  quality 
product,  at  predictabie  cost,  and  in  accordance  with  an  established  schedule. 

DON  Software  Organization:  A  DON  organization  or  subdivision  thereof,  that  acquires,  de¬ 
velops  or  maintains  software,  software  products,  or  software  work  products.  It  includes  Central 
Design  Activities  (CDAs),  Software  Support  Activities  (SSAs),  Warfare  and  Warfare  Support 
Systems  (WWSS)  activities  as  well  as  activities  in  which  base  level  computing  (BLC)  is  per¬ 
formed.  It  applies  to  all  organizations  that  manage  software  related  activities  under  DoD 
5000.1,  DoD  8000.1 ,  DoD  Instruction  5000.2,  and  applicabie  Navy  directives/instructions. 

Software  Personnel:  All  personnel  that  are  involved  with  software  development  and  mainte¬ 
nance:  managers,  engineers,  designers,  programmers,  coders,  testers,  software  quality  as¬ 
surance  and  configuration  management  personnel.  It  does  not  include  clerical  or 
administrative  personnel. 

Software  Product:  Software  or  associated  information  created,  modified,  or  incorporated  to 
satisfy  a  contract.  Examples  include  plans,  requirements,  design,  code,  databases,  test  infor¬ 
mation  and  manuals. 

Software  Work  Product:  Any  artifact  created  as  part  of  defining,  maintaining,  or  using  a  soft¬ 
ware  process,  including  process  descriptions,  plans,  procedures,  computer  programs,  and  as¬ 
sociated  documentation,  which  may  or  may  not  be  intended  for  delivery  to  customer  or  end 
user.  j 

I 

Source  Line  of  Code  (SLOC):  All  prograrh  instructions  created  by  project  personnel  or  other 
automated  method  which  are  then  processed  into  machine  code  by  some  combination  of  pre¬ 
processors,  compilers  and  assemblers.  It  includes  declarative  and  executable  statements  as 
well  as  job  control  language  .  It  does  not  include  comments,  and  is  measured  at  one  state¬ 
ment  per  line  of  code.  To  estimate  expected  SLOC,  a  process/method/tool/technique  which 
is  approved  by  the  organization  should  be  used. 

Unprecedented  systems:  A  system  is  deemed  unprecedented  if  it  does  not  meet  one  or 
more  of  the  following  criteria; 

a.  the  requirements  are  consistent  and  well-understood; 

b.  the  system  architecture,  both  hardware  and  software,  known  to  be  adequate  f 
or  the  requirements; 

c.  the  acquisition  and  development  teams  have  previously  developed  a  similar 
system. 
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1.  Background.  For  the  purpose  of  this  policy,  the  terms  assessment  and  SPA  shall  be 
used  to  designate  an  internal  appraisal  of  an  organization  based  upon  the  CMM.  The 
terms  evaluation  and  SCE  shall  be  used  to  designate  an  external  appraisal  of  an  organi¬ 
zation  based  upon  the  C  MM .  All  appraisal  methods  will  be  based  upon  the  CMM  and  shall 
be  approved  for  use  by  DON.  Implementation  guidance  for  each  is  described  in  more  de¬ 
tail  below. 

2.  Software  Process  Improvement  (SPI).  Software  process  improvement  begins  with  an 
assessment.  The  finding  and  recommendations  from  that  assessment  form  the  basis  for 
an  action  plan/software  process  improvement  (SPI)  plan. 

a.  A  software  organization  should  contact  their  primary  DON  SPI  Center  of 
Excellence  to  obtain  information  on  assessments  and  on  software  process 
improvement.  The  Center  of  Excellence  will  be  available  to  assist  them  in 
improving  their  software  processes  . 

b.  It  is  recommended  that  an  interim  improvement  progress  check  should  be 
made  at  no  more  than  18  month  intervals.  At  this  time  the  SPI  Plan  should  be 
revised  if  necessary. 

The  goal  of  the  DON  SPI  program  is  to  improve  the  organization's  ability  to  develop  and  main¬ 
tain  software  as  measured  periodically  against  the  SEI  CMM  model . 

3.  CMM  Based  Software  Process  Appraisals.  The  Capability  Maturity  Model  (CMM)  was 
developed  at  the  Software  Engineering  Institute  (SEI) .  The  CMM  is  a  structured  model 
which  describes  five  maturity  levels  of  a  software  organization  and  identifies  Key  Process 
Areas  (KPAs)  within  each  level.  The  SEI  Software  Process  Assessment  (SPA)  methodol¬ 
ogy  has  gained  widespread  industry  and  DoD  acceptance  as  a  method  for  establishing  the 
software  process  maturity  of  an  organization.  Recently  the  SEI  began  the  process  of  align¬ 
ing  the  CMM  based  Software  Process  assessments  (SPA)  and  the  Software  Capability 
Evaluations  (SCEs).  They  both  fall  under  the  CMM-Based  Appraisal  umbrella.  The  new 
SPA  method  is  called  CMM-Based  Appraisal  Internal  Process  Improvement  (CBA  IPI). 
The  SCE  method  is  still  called  an  SCE  under  the  new  CBA  umbrella. 
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a.  The  DON  SPI  Center  of  Excellence  will  be  the  primary  source  of  trained 
assessors  and  of  assistance  for  software  organizations  within  their  customer 
base.  They  will  be  available  to  assist  them  in  determining  their  assessment 
needs. 

b.  Assessments  will  be  conducted  by  a  team  composed  of  personnel  from  the 
assessed  software  organization  and  trained  assessors.  The  assessment  team 
will  prepare  a  written  report  containing  areas  or  improvement  and 
recommendations.  The  report  will  be  written  so  improvement  areas  are  not 
attributed  to  any  project  or  person  associated  with  the  project.  No  other  agency 
will  receive  the  report. 

c.  Small  software  sites  (under  75  software  personnel)  may  request  a  less 
personnel  and  dollar  resource  intensive  assessment,  based  upon  the  SEI 
CMM. 

d.  The  results  of  the  assessments  will  be  held  in  strictest  confidence.  The 
assessment  team  prepares  the  written  report  containing  areas  of  improvement 
and  recommendations.  The  report  is  written  so  improvement  areas  are  not 
attributed  to  any  project  or  people  associated  with  the  project.  No  other  agency 
or  activity  receives  the  report  without  the  consent  of  the  assessed  organization. 

4.  Software  Capability  Evaluation  (SCE).  The  SCE  is  a  structured  methodology  developed 
by  the  SEI  to  conduct  an  evaluation  of  an  offeror's  software  process  capability .  The  SCE 
shall  serve  as  the  basis  of  a  software  risk  assessment  and  shall  provide  an  objective 
means  for  assessing  an  offeror's  software  process  capabilities.  An  SCE  evaluates  the 
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contractor’s  software  engineering  processes,  and,  based  on  the  evaluation,  determines 

the  strengths  and  weaknesses  associated  with  the  Key  Process  Areas  of  the  CMM.  The 

degree  of  risk  can  be  determined  from  analysis  of  the  strengths  and  weaknesses. 

a.  Assistance  in  defining  and  performing  the  SCEs,  to  include  RFP  preparation, 
should  be  obtained  from  the  primary  SPI  Center  of-Excellence.  If  the  primary 
SPI  Center  of  Excellence  is  unable  to  meet  the  schedule  constraints,  other 
activities  may  be  used. 

b.  Intent  to  use  SCEs  shall  be  inserted  in  Section  L  or  M  of  the  Request  for 
Proposal  (RFP).  The  RFP  should  state  explicitly  that  the  SCE  will  be 
accomplished  by  government  personnel  during  the  source  selection.  It  should 
also  state  that  the  SCE  team  may  be  separate  and  distinct  from  the  proposal 
evaluating  team. 

c.  The  Instructions  for  Preparation  of  Proposals  shall  identify  the  documentation 
requested  by  the  evaluation  team,  such  as  project  profiles,  organization  charts, 
sample  documentation,  and  a  software  process  improvement  plan.  It  shall  also 
request  the  offeror  to  provide  the  SCE  team  with  facilities  during  the  site  visit. 

d.  The  SCEs  shall  be  conducted  by  trained  government  teams.  To  ensure 
consistency  in  the  application  of  the  evaluation  methodology,  it  is 
recommended  that  the  same  team  evaluate  all  offerors  for  an  acquisition.  If  the 
same  team  is  unable  to  complete  all  the  SCEs,  the  Program  Manager  will 
discuss  the  impacts  of  that  decision  with  the  legal  counsel  for  the  cognizant 
contracting  office. 

e.  The  results  of  the  pre-award  evaluations  shall  be  used  in  conjunction  with 
results  of  other  technical  evaluations  performed  by  the  source  selection 
evaluation  board  (SSEB).  The  intent  of  the  evaluation  is  to  determine  program 
risk  associated  with  the  offeror’s  software  process  capability  and  should  not  be 
used  to  limit  competition  to  contractors  that  may  satisfy  a  predefined  level  of 
software  process  maturity.  The  results  should  be  planned  to  be  used  only  in  the 
acquisition  for  which  they  were  accomplished.  Use  of  previously  accomplished 
SCEs  is  strongly  discouraged  but,  if  they  are  used,  the  Program  Manager  will 
discuss  the  impacts  of  that  decision  with  the  legal  counsel  for  the  cognizant 
contracting  office. 

f.  It  is  recommended  that  an  SCE  be  conducted  on  each  prime  contractor  within 
the  competitive  range.  The  SCE  should  take  place  at  the  site  where  the  majority 
of  the  critical  software  is  being  developed  or  maintained. 

g.  Conducting  SCEs  on  Subcontractors.  The  Government  shall  reserve  the  right 
to  conduct  software  process  risk  evaluations  (SCEs)  on  all  subcontractors. 
Selection  of  the  subcontractors  for  evaluation  should  be  based  upon  Program 
Manager  specified  criteria.  This  criteria  shall  identify  subcontractors  who 
contribute  significantly  to  the  program  risk.  This  criteria  may  include  not  only 
the  amount  of  software  code  or  components  developed  but  also  contributions 
such  as  configuration  management,  software  quality  control,  software  test, 
software  design  and  software  documentation.  If  it  is  determined  that 
subcontractors  are  to  receive  SCEs,  then  the  visit  should  be  at  the  invitation  of 
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the  prime  and  with  the  participation  of  the  prime.  It  is  recommended  that  the 
SCE  take  place  at  the  subcontractor's  site.  It  is  preferable,  however,  that  the 
prime  contractor  conduct  the  SCE  on  the  subcontractor  and  provide  results  to 
the  government  team. 

h.  RFP  Modifications.  During  contract  award  negotiations,  and  based  upon  the 
results  of  the  evaluation,  the  DON  program  manager  may  elect  to  significantly 
tailor  the  requirements  for  software  documentation  and  frequency  of  software 
program  reviews  originally  specified  in  the  RFP.  The  intent  to  do  so  should  be 
clearly  stated  in  the  RFP  and  is  necessary  to  maintain  a  fair  open  competition 
among  all  contractors  at  various  levels  of  the  maturity  model .  The  intent  is 
to  manage  by  risk  assessment  and  not  by  level  of  maturity. 

i.  Broad  based  evaluations.  The  DON  Program  manager  is  expected  to  conduct 
risk  evaluations  that  measure  a  full  range  of  the  offeror's  software 
development/maintenance  capabilities. 

j.  Contract  Incentives .  The  DON  program  manager  is  encouraged  to  incentivize 
the  contractor  to  establish  a  software  process  improvement  program  that  will 
mitigate  areas  of  risk  identified  in  the  evaluation  . 

k.  Post-award  SCEs.  The  DON  program  manager  is  encouraged  to  conduct 
periodic  post  award  evaluations  as  a  risk  management  tool  to  ensure  the 
contractor  maintains  its  software  process  capability  and  does  not  allow  a 
deterioration  of  process  that  may  introduce  a  program  risk  not  identified  during 
the  source  selection  evaluation.  Post-award  SCEs  have  been  used  to 
determine  an  award  fee  based  upon  a  SPI  plan.  Acquisition  officials  have  used 
an  SCE  in  conjunction  with  a  value  engineering  incentive  clause  to  provide  a 
method  for  claiming  cost  savings.  SCEs  have  also  been  used  without  incentive 
as  a  process  improvement  oversight  tool. 

l.  DON  Program  Executive  Officers  (PEOs),  Direct  Reporting  Program 
Managers  (DRPMs)  and  program  Managers  (PMs)  will  include  provisions  for 
SCE  planning  in  their  Computer  Resource  Life  Cycle  Management  Plan 
(CRLCMP)  or  Software  Management  Plan.  During  the  development  of  the  RFP 
and  acquisition  strategy,  they  are  responsible  to  fully  coordinate  with  their 
supporting  DON  SPI  Center  of  Excellence  the  schedule  and  budget  for 
resources  necessary  to  implement  this  SPI  Policy. 

m.  Alternative  Evaluation  Methodology.  The  SEI  CMM  based  SCE  method  is 
required  by  this  policy.  This  does  not  preclude  the  selection  and  utilization  of 
comparable  models  and  evaluation  methodologies  at  some  future  time,  it 
recognizes  the  dynamic  evolution  of  software  engineering  technology  and  the 
existence  of  other  models  that  are  not  specifically  based  upon  the  SEI  model. 
These  include  the  European  Scientific  Project  on  Information  Technology's 
(ESPRIT)  Bootstrap,  SCOPE,  and  the  draft  International  Standards 
Organization  (ISO)  Software  Process  Improvement  Capability  Determination 
(SPICE)  method.  The  CMM  based  SCE  method  is  expected  to  move  toward 
the  ISO  SPICE  method  when  it  is  released. 
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Department  of  the  Air  Force 

Headquarters  56th  Air  Base  Wing  (AFMC) 
Hanscom  Air  Force  Base,  Massachusets 


2  February  1 995 


Memorandum  for  Distribution  F 

From:  ESC/CD 

Subject:  Software  Capability  Evaluation  Policy 

1 .  ESC  is  committed  to  developing  quality  software-intensive  programs  that  meet  mission 
needs  on-time  and  within  budget  Key  to  the  success  of  these  acquisitions  is  an  evaluation 
of  the  developer's  software  capability.  SAF/AQ  issued  Acquisition  Policy  94A-009,  dated 
23  Aug  94,  requiring  software  development  capability  evaluations  for  software  intensive 
systems. 

2.  This  memo  establishes  ESC's  Policy,  in  compliance  with  SAF/AQ.  Software  intensive  MIS 
and  C3I  System  acquisitions  at  ESC  will  use  the  Software  Engineering  Institute  (SEI)  Soft¬ 
ware  Capability  Evaluation  (SCE).  The  results  of  these  evaluations  will  serve  as  an  input 
in  the  overall,  source  selection  evaluation  process. 

3.  The  ESC  Software  Center  (ESC/ENS)  is  the  point  of  contact  for  all  SCE  team  visits  and 
team  structure.  Our  newly  established  Acquisition  Support  Office  (PKA)  is  the  point  of 
contact  for  all  action  pertaining  to  the  development  of  evaluation  criteria  and  standards  and 
for  documentation  developed  throughout  the  source  selection  process.  Recognizing  that 
today's  program  offices  do  not  have  sufficient  expertise  available  to  do  these  evaluations 
and  source  selections  concurrently,  ENS  has  established  a  contract  to  advise  and  assist 
program  offices  in  conducting  SCEs.  Guidelines  for  implementing  this  policy  are  attached. 

4.  For  assistance  in  determining  whether  SCE  requirements  should  be  incorporated  into  your 
RFP  and  in  formulating  an  in-plant  team  to  conduct  SCE's  contact  ESC/ENS.  Ms  Kathleen 
McCullough  3-8493  or  Ms  Cathi  Sparaco  3-8491  or  Fax  3-8325.  For  assistance  in  struc¬ 
turing  your  source  selection  evaluation  criteria  and  standards  and  for  preparing  associated 
reports/briefings/debriefing,  contact  ESC/  PKA,  3-5852,  Fax  3-9959. 

PHILIP  P.  PANZARELLA,  SES 
Executive  Director 


Attachment: 

SCE  Policy  Implementation  Plan 


CMU/SEI-95-TR-012 


©1995  Integrated  System  Diagnostics,  Inc.  and  Carnegie  Mellon  University 


173 


Current  DoD  Policy  Directives:  SCE 


April  1996 


Software  Capability  Evaluation  Policy 

Implementation  Plan 

1 .  Purpose.  This  Implementation  Plan  provides  guidelines  for  incorporating  Software  Capa¬ 
bility  Evaluations  (SCE)  into  the  source  selection  process  at  the  Electronic  Systems  Cen¬ 
ter. 

2.  Objective.  An  SCE  is  an  independent  evaluation  of  an  offeror's  software  process  at  the  lo¬ 
cation  where  the  offeror  proposes  to  accomplish  the  predominance  of  the  software  devel¬ 
opment  effort.  It  is  a  tool  that  can  help  a  Program  Office  to  determine  an  offeror's  ability  to 
produce  a  high  quality  product  on  time  and  within  budget.  The  objective  of  the  SCE  is  to 
identify  the  strengths,  weaknesses,  and  existing  improvement  activities  in  an  offeror's  soft¬ 
ware  process  that  best  indicate  the  risk  associated  with  using  that  offeror  for  a  particular 
software  acquisition.  SCE  results  provide  one  input  to  the  overall  source  selection  evalu¬ 
ation  process. 

3.  References 

a.  SAF/AQ  Policy  Memorandum  94A-009,  "Use  of  Software  Development 
Capability  Evaluation  in  Source  Selections",  Washington,  D.C.,  Aug.  1994. 

b.  Capability  Maturity  Model  for  Software,  Version  1.1,  CMU/SEI-93-TR-24, 
Software  Engineering,  Institute,  Carnegie  Mellon  University,  Pittsburgh,  PA, 

Feb.  1993  (DTIC  Report  ADA  240603) 

c.  Software  Capability  Evaluation  (SCE)  Implementation  Guide,  Version  2.0, 

CMU/  SEI-94-TR-05,  Software  Engineering  Institute,  Carnegie  Mellon 
University,  Pittsburgh,  PA,  Feb.  1994  (DTIC  Report  ADA  240604) 

4.  Background.  The  Software  Engineering  Institute  (SEI)  was  established  in  1 984  to  address 
the  Nation's  growing  software  problems:  weapon  system  schedule  slips  due  to  software: 
unsatisfied  system  requirements:  system  failures  due  to  latent  software  defects:  and  cost 
overruns  due  to  software.  Over  the  past  decade,  the  continued  rapid  growth  in  the  size, 
cost,  complexity,  and  functionality  of  software  in  military  systems  has  exacerbated  the  soft¬ 
ware  crisis.  To  help  alleviate  the  crisis,  the  SEI  developed  a  process  maturity  framework 
which  organizations  could  use  to  improve  their  software  development  process.  The  Capa¬ 
bility  Maturity  Model  (CMM),  released  in  1 993,  provides  organizations  guidance  for  estab¬ 
lishing  process  improvement  programs  and  is  the  basis  for  improving  their  overall  software 
process.  The  CMM  is  based  on  the  premise  that  the  quality  of  a  product  depends  upon  the 
quality  of  the  process  used  to  create  it.  The  CMM  framework  describes  an  evolutionary 
path  from  ad  hoc,  chaotic  software  processes  to  mature,  disciplined  software  processes. 
The  SCE  provides  a  method,  based  upon  the  CMM,  to  evaluate  an  organization's  soft¬ 
ware  process,  i.e.,  the  strengths  and  weaknesses,  to  help  determine  the  degree  of  risk  as¬ 
sociated  with  its  software  development  capability.  The  SCE  Implementation  Guide  pro¬ 
vides  Program  Managers  guidance  for  using  the  SCE  method  during  an  acquisition.  In 
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June  1993,  SAF/AQ  first  directed  that  SCEs  be  used  in  Air  Force  source  selections  for  Man¬ 
agement  Information  Systems  and  C3I  Systems. 

5.  Scope.  Program  Offices  shall  include  SCEs  in  the  source  selection  process  for  all  soft¬ 
ware-intensive  systems,  or  modifications  to  existing  software-  intensive  systems,  if  any  of 

the  following  conditions  exist: 

a.  Software  is  critical  to  the  system's  mission  accomplishment 

b.  Software  constitutes  a  major  portion  of  the  overall  development  effort. 

c.  A  major  component  of  the  system,  including  its  software  functionality,  is 
unprecedented. 

d.  Software  development  costs  exceed  $5  Million. 

e.  Software  developed  during  a  demonstration/validation  phase  is  used  in  a 
follow-on  contract  phase. 

6.  General  Procedures. 

a.  Program  Offices  for  Management  Information  and  C3I  Systems  shall  conduct 
SCEs  in  accordance  with  the  SEI-developed  CMM  and  Implementation  Guide 
when  any  of  the  conditions  in  Paragraph  5  apply. 

b.  Program  Managers  shall  be  responsible  for  developing  acquisition  strategies 
which  include  provisions  for  SCEs.  The  use  of  SCEs  during  source  selections 
must  be  specified  in  the  Request  for  Proposal  (RE:P).  The  SCE  method  should 
not  be  altered:  however,  use  of  the  SCE  findings  may  be  tailored  for  individual 
source  selections.  The  source  selection  decision  shall  not  be  based  solely  on 
the  SCE  findings. 

c.  Unless  award,  without  discussions  is  appropriate,  SCE  s  will  be  conducted  on 
all  offerors  in  the  competitive  range.  Government  only  teams  or  government 
teams  supported  by  an  SCE  Support  Services  Contractor,  trained  in  the  SCE 
method,  shall  conduct  the  evaluations  at  offeror  locations  where  the 
predominance  of  software  development  is  anticipated.  To  insure  consistency  in 
the  application  of  the  SCE  methodology,  the  same  SCE  Team  shall  evaluate  all 
offerors  for  an  acquisition. 

d.  The  SCE  findings  shall  be  incorporated  into  the  overall  source  selection 
evaluation.  The  findings  will  identify  each  offeror's  CMM-related  strengths, 
weaknesses,  and  process  improvement  activities.  The  Contracting  Office  . 
should  include  the  successful  offeror's  established  software  process  and 
planned  improvements  in  the  contract. 

7.  Specific  Responsibilities. 

a.  ESC  Program  Offices: 

(1)  Coordinate  with  ESC/ENS  to  determine  the  need  for  an  SCE  and 
to  plan  for  all  follow-on  SCE  activities. 
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(2)  Discuss  the  program’s  requirement  for  an  SCE  at  the  Strategic 
Round  Table  and  the  Acquisition  Strategy  Panel. 

(3)  Include  provisions  for  the  SCE  in  all  appropriate  acquisition 
documents:  Commerce  Business  Daily,  Acquisition  Strategy  Plan, 
Source  Selection  Plan,  Evaluation  Criteria,  and  RFP  Sections  H,  M, 
and  L  (Instructions  for  Proposal  Preparation). 

(4)  Determine  whether  to  establish  an  all  Government  SCE  Team  or 
to  obtain  contractor  support  through  the  SCE  Support  Service 
Contract  in  establishing  a  joint  Government/Contractor  team. 
Determine  SCE  Team  composition  and  training  needs. 

(5)  Determine  funding  requirements  for  SCE  Team  activities  to 
include  training  and  travel.  Coordinate  with  ESC/ENS  to  prepare 
Delivery  Orders  for  SCE  contractor  support,  if  required. 

(6)  Coordinate  SCE  Team  activities  and  site  visits. 

(7)  Present  SCE  findings  to  the  Source  Selection  Evaluation 
Board/Team,  Advisory  Council,  and  Authority. 

(8)  Insure  that  SCE  findings  are  presented  in  all  post  award 
debriefings  to  the  offerors. 


b.  ESC/PK: 


(1)  Insure  that  the  Strategic  Round  Table  and  Acquisition  Strategy 
Panel  briefings  address  this  SCE  policy  and  include  an  SCE 
determination  by  each  Program  Office 

(2)  Assists  program  office  in  the  development  of  appropriate  RFP 
language,  e.g.  Section  M  evaluation  criteria;  Section  L  (IFPP),  etc.  in 
conjunction  with  ESC/ENS. 

(3)  When  needed  provide  Software  specialist  to  assist  SCE  team  in 
conduct  of  SCE’s  at  offeror  locations. 

(4)  Insure  that  all  source  selection  documentation  addresses  the  use 
of  SCE's  and  that  the  results  are  reflected  in  the  evaluation/analysis 
reports/briefings  and  source  selection  decision  documents,  as 
appropriate. 

c.  ESC  Software  Center  (ENS): 

(1 )  Be  the  ESC  advocate  for  SCEs. 

(2)  Provide  SCE  advice  and  support  to  ESC  Program  Offices  in 
conjunction  with  ESC/PKA's  Software  Specialist. 
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(3)  Assist  Program  Offices  in  determining  training,  travel,  and  funding 
needs. 

(4)  Manage  the  SCE  Support  Services  Contract 

(5)  Assist  Program  Offices  in  the  preparation  of  Delivery  Orders  for 
SCE  contractor  support,  when  required. 
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DEPARTMENT  OF  THE  AIR  FORCE 
WASHINGTON  DC 

Acquisition  Policy  94A-009 

MEMORANDUM  FOR  DISTRIBUTION  Aug  23  1994 

FROM:  SAF/AQ 

1060  Air  Force  Pentagon 
Washington,  DC  20330-1060 

Subject:  Use  of  Software  Development  Capability  Evaluation  in  Source  Selections. 
This  policy  memo  replaces  AQ  Policy  Memo  93M-003 

The  Air  Force  is  committed  to  improving  the  acquisition,  development,  and  support 
processes  associated  with  software-intensive  systems.  Critical  to  the  success  of  these  acqui¬ 
sitions  is  the  evaluation  of  potential  developers’  capability  to  deliver  quality  software  at  a 
predictable  cost  and  in  accordance  with  an  established  schedule.  Accordingly,  software 
development  capability  evaluations  shall  be  regularly  used  in  conjunction  with  source  selec¬ 
tions  for  software  intensive  systems. 

The  two  software  development  capability  tools  authorized  for  use  in  air  Force  source 
selection  evaluations  (with  implementation  guidance  at  Attachment  2)  are: 

a.  For  Management  Information  Systems  and  Command,  Control,  Computer,  and  Intelli¬ 
gence  Systems,  use  the  Camegie-Mellon  University  (CMU)  Software  Engineering  Institute 
(SEI)  method  based  on  the  SEI  Capability  Maturity  Model  as  defined  in  the  Software  Capa¬ 
bility  Evaluation,  Version  2.0,  Implementation  Guide,  CMU/SEI-94-TR-05.  Assistance  in 
defining  and  performing  SEI  method  capability  evaluations  in  conjunction  with  source 
selection  activities  may  be  obtained  from  the  HQ  Electronics  Systems  Center  Software  Cen¬ 
ter  (ESC/ENS)  at  DSN  478-8561  or  commercial  (617)  377-8561. 

b.  For  Weapon  System  (embedded  software)  applications,  and  wherever  systems  engi¬ 
neering  is  the  predominate  management  consideration  due  to  the  interaction/integration  of 
hardware  and  software  environments,  use  the  Aeronautical  Systems  Center  (ASC)  Software 
Development  Capability  Capacity  Review  (SDCCR)  as  defined  in  ASC  Pamphlet  8(X)-5. 
Assistance  in  defining  and  performing  the  SDCCR  may  be  obtained  form  HQ  ASC  Embed¬ 
ded  Computer  Resource  Program  Office  (ASC/EN(CR)  at  DSN  785-3656  or  commercial 
(513)  255-3656. 
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OPR  for  this  policy  memorandum  is  SAF/AQKS  at  ADSN  227-3108  or  commercial  (703) 
697-3108. 

2  Attachments 

1 .  Distribution  List 

2.  Implementation  Guidance 
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AFMC/CC/XR/XPD 
ASC/CC 
ESC/CC 
HSC/CC 
SMC/CC 
00-ALC/CC 
SA-ALC/CC 
SA-ALC/CC 
SM-ALC/CC 
WR-ALC/CC 
AFMX/XPD 
Program  Managers 
System  Program  Directors 
SAF/AQC/AQK/AQL/AQP/AQQ/AQS/AQT/AQX 
SAF/FMC 
SA/IAO 
SAV/SN/SX 

PEO/C/  / 

/CM 

/Cl 

/ST 

/SP 

/TA 

/TS 

AF/IN 

AF/LG/LGM 

AF/PE 

AF/RER 

AF/SE/SEP 

AF/SC 

AF/TE/TEP/TER 

AF/XO/XOR 

AFC4A/CC 

AFOTEC/CC/XRX 

AFSPC/CC 

AFSPC/DRR 
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AETC/CC 

AFIC/LEA 

AMC/CC 

ACC/CC 

USAFE/LGCG 

AFIA/AI 

AFAA/CC/AG/QLP/QLW 

AFCAA/CC 

AFIT/LSY 

DSMC 

SCMC-DD 

NAVIR/PMA  242-1 1C  Attachment  1 

Attachment  2  -  Implementation  Guidance 

1 .  The  SEI  method  based  on  the  SEI  Capability  Maturity  Model  defined  in  CMU/sEI-87- 
TR-23  evaluates  the  contractor’s  software  engineering  processes  and,  based  on  the  evalua¬ 
tion,  determines  the  strengths  and  weaknesses  associated  with  key  process  areas  of  the 
Capability  Maturity  Model.  The  degree  of  risk  can  be  determined  from  the  strengths  and 
weaknesses. 

2.  The  Aeronautical  Systems  Center  Software  Development  Capability  Capacity  Review 
(SDCCR)  method  defined  in  ASC  Pamphlet  800-5  ev^uates  their  contractors’  capability 
and  capacity  to  develop  software  within  the  context  of  the  overall  system  development  and 
includes  coverage  of  the  systems  engineering  development  capability  as  well  as  other  pro¬ 
cesses  and  disciplines  related  to  the  software  development. 

3.  The  objective  of  these  evaluations  is  to  provide  a  structured,  consistent,  and  comprehen¬ 
sive  approach  for  evaluating  the  software  process  to  determine  the  software  development 
capability  of  organization(s)  with  primary  software  development  responsibilities  under 
planned  contracts. 

4.  One  of  the  two  software  development  capability  evaluation  methods  authorized  above 
shall  be  employed  during  the  source  selection  process  on  all  software-intensive  systems,  and 
major  modifications  to  existing  software-intensive  systems,  if  any  of  the  following  condi¬ 
tion  is  met: 

a.  The  software  is  critical  to  the  system’s  mission  accomplishment. 

b.  Software  constitutes  a  major  portion  of  overall  development  effort. 

c.  A  major  component  of  the  system,  including  its  software  functionality,  is  considered  to 
be  unprecedented. 

d.  Software  development  cost  will  exceed  $5  Million. 
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e.  Software  developed  during  a  demonstration/validation  effort  is  planned  to  be  used  dur¬ 
ing  a  follow-on  contract  phase. 

5.  The  Program  Manager  will  be  responsible  for  developing  an  acquisition  strategy  which 
provides  for  use  of  software  development  capability  evaluation  as  required  above.  Use  of 
the  findings  in  the  source  selection  (as  risk  determinant,  technical  or  management  factor) 
may  be  tailored  for  each  acquisition.  The  source  selection  decision  will  not  be  based  solely 
on  the  capability  evaluation  findings.  The  findings  may  be  used  during  source  selection  if 
this  intention  is  stated  in  the  RFP.  (Note:  The  process  for  determining  each  offeror’s  capabil¬ 
ity  shall  conform  to  the  ESC  or  ASC  methods  described  above.) 

6.  Software  development  capability  evaluations  performed  using  the  SEI  method  shall  be 
conducted  at  one  or  more  contractor  locations/organizations  where  the  predominance  of 
vital  software  will  be  developed.  Software  development  capability  evaluations  performed 
using  the  SDCCR  shall  also  be  conducted  at  one  or  more  contractor  locations/organizations 
where  the  predominance  of  vital  software  will  be  developed  except  for  those  cases  when  a 
contact  is  awarded  without  discussions.  For  those  cases,  the  capability  evaluation  will  be 
based  on  an  assessment  of  the  offeror’s  proposal  response  to  the  SDCCR  requirements.  All 
evaluations  shall  be  conducted  by  trained  government  teams.  To  ensure  consistency  in  the 
application  of  the  evaluation  methodology,  the  same  team  shall  evaluate  all  offerors  for  an 
acquisition. 

7.  Software  development  capability  evaluation  findings  will  be  developed  by  or  summarized 
and  presented  to  the  Source  Selection  Evaluation  Board  (SSEB)  or  Source  Selection  Evalu¬ 
ation  Team  (SSET)  for  incorporation  into  the  overall  source  selection  evaluation.  The  find¬ 
ings  will  detail  each  offeror’s  strengths,  weaknesses  and  risks  and  will  also  identify  any 
process  improvement  activities  undertaken  or  planned  by  the  contractor.  The  results  from 
these  software  development  capability  evaluations  will  be  used  only  in  the  acquisition  for 
which  they  were  accomplished  and  will  not  be  disclosed  by  the  government  for  any  other 
purpose  without  the  offeror’s  permission. 

8.  The  program  office  will  incorporate  the  offeror’s  process,  and  planned  improvements, 
into  relevant  portions  of  the  model  contract. 
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Acquisition  Start 


Organize /Select 
SCE  Team 


Execute 

Acquisition  Start 
Phase 


SCE  Implementation  Checklist 

□  Develop  initial  awareness 

n  Determine  applicability  to  this  acquisition 

□  Review  existing  SCE  policies  and  procedures 

□  Review  acquisition  strategy 

□  Determine  SCE  needs  for  acquisition 

□  Develop  SCE  implementation  recommendation 

□  Input  to  acquisition  strategy  document 

□  Obtain  commitment  to  use  SCE 

□  Review  SCE  team  leader  and  team  member  qualification 
criteria 

□  Ensure  appropriate  criteria  for  team  are  applied  to  acquisition 

□  Prepare  candidate  SCE  team  member  list 

□  Obtain  commitment  from  candidate  team  member’s 
organization 

□  Familiarize  team  with  acquisition  policies  and  procedures 

□  Attend  SpE  training 

□  Determine  SCE  placement  within  source  selection 
documentation 

□  Prepare  recommendation  on  how  SCE  findings  will  be 
integrated  into  the  acquisition 

□  Develop  Product  Profile 

□  Determine  Target  Process  Capability  (TPC) 

□  Determine  disposition  of  SCE  data 

□  Estimate  number  of  contractor  sites  to  be  visited 

□  Estimate  resources  and  time  (manpower,  travel,  support) 

□  Determine/schedule/implement  preliminary  SCE  tasks 

□  Complete  CBD  announcement  input 

□  Prepare  Pre-proposal  Conference  Briefing  (if  applicable) 
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Execute  General 
and  Specific 
Preparation 
Phases 


Conduct  SCE 


□  Insert  Acquisition  Plan,  Source  Selection  Plan,  RFP  SCE 
language 

□  Request  completion  of  Maturity  Questionnaire  and  product 
profiles 

□  Instructions  on  how  to  submit  material 

□  Prepare  Evaluation  Standards 

□  Schedule  SCE  team  to  meet  and  execute  SCE  Method 
pre-site  visit  preparation. 

□  Analyze  product  profiles 

□  Select  contractor  projects 

□  Prioritize  process  areas  for  all  contractors 

□  Determine  key  issues  for  individual  contractors 

□  Develop  initial  interview  questions  and  identify  initial  set  of 
documents  for  review 

□  Develop  and  notify  contractor  points  of  contact  regarding  SCE 
team  logistical  requirements  (10  working  days  in  advance) 

□  Arrange  site  logistics  (room,  table,  chairs,  documents, 
preliminary  on-site  and  interview  schedules,  computing  needs, 
etc.) 

I 

□  Conduct  SQE  site  data  collection 

□  Conduct  in-briefing  with  on-site  contractor 

□  Analyze  organizational  and  project  documentation 

□  Review  and  modify  agenda  and  schedule  as  necessary 
n  Conduct  initial  interviews 

□  Request  additional  documentation 

□  Validate  interview  responses 

□  Prepare  draft  findings 

□  Validate  draft  findings 

□  Conduct  consolidation  interviews 

□  Validate  improvement  activities 

□  Develop  final  findings 
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□  Conduct  exit  briefing  (as  prescribed  by  Procuring  Contracting 
Officer  [PCO]) 


Write  /  Submit 
Final  Report  to 
Acquisition 
Organization 


□  Document  conduct  of  SCE  and  rationale  for  findings 

□  Document  effort  and  resources  expended 

n  Develop  lessons  learned  and  provide  feedback  to  improve  SCE 
Method 


Assist 
Acquisition 
Organization’s 
Use  of  SCE 
Findings 


□  Develop  and  deliver  final  SCE  results  briefing  to  SSEB  (if 
necessary) 

□  Consult  with  SSEB  and  SSAC  as  needed  (elaborate  on  SCE 
findings) 

□  Assist  SSEB  in  preparing  and  delivering  formal  SCE 
presentation  to  the  SSAC 


Formal 

Feedback 


□  Conduct  SCE  findings  briefing  for  winning  contractor 

□  Conduct  SCE  findings  briefing  for  unsuccessful  offerors 

□  Dispose  of  SCE  data  (in  accordance  with  acquisition  guidelines) 

□  Disband  SCE  team 
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Appendix  F 


Acronyms 

AMC 

Army  Materiel  Command 

AMIS 

Acquisition  Management  Information  System 

BAFO 

Best  and  Final  Offer 

CAO 

Contract  Administration  Office 

CBD 

Commerce  Business  Daily 

CDR 

Critical  Design  Review 

CDRL 

Contract  Data  Requirements  List 

CMM 

Capability  Maturity  Model 

CPAR 

Contractor  Performance  Analysis  Report 

CPEP 

Contractor  Performance  Analysis  Program 

CRs 

Clarification  Requests 

CSC 

Computer  Software  Component 

CSCI 

Computer  Software  Configuration  Item 

CSU 

Computer  Software  Unit 

DCAA 

Defense  Contracting  Audit  Agency 

DoD 

Department  of  Defense 

OTIC 

Defense  Technical  Information  Center 

DemA/al 

Demonstration/Validation 

DRs 

Deficiency  Reports 

DSMC 

Defense  Systems  Management  College 

EMD 

Engineering  Manufacturing  Development 

EP 

Evaluation  Plan 

ESC 

Electronic  Systems  Center 

FAR 

Federal  Acquisition  Regulation 

FCA 

Functional  Configuration  Audit 
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GAO 

IFPP 

IRS 

JPO 

JTIDS 

KPA 

KSLOC 

LTR 

MCCR 

MMP/CR 

MQ 

NAWC 

NRAD 

NTE 

PCA 

PCO 

PDR 

PEO 

PFN 

PM 

POC 

PRAG 

RAI 

RFP 


General  Accounting  Office 
Instructions  for  Proposal  Preparation 
Interface  Requirements  Specification 
Joint  Program  Office 

Joint  Tactical  Information  Distribution  System 

Key  Process  Area 

Thousand  Source  Lines  of  Code 

Letter 

Mission  Critical  Computer  Resources 

Manufacturing  Management  Production/Capability 
Review 

Maturity  Questionnaire 
Naval  Air  Warfare  Center 

NCCOSC  (Naval  Command,  Control,  and  Ocean  Sur¬ 
veillance  Center)  RDT&E  (Research  Development 
Test  and  Engineering)  Division 

Not  to  Exceed 

Physical  Configuration  Audit 

Procuring  Contracting  Officer 

Preliminary  Design  Review 

Program  Executive  Officer 

Point  For  Negotiation 

Program  Manager 

Point  Of  Contact 

Performance  Risk  Analysis  Group 
Request  for  Additional  Information 
Request  For  Proposal 
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Acronyms 


SCE 

Software  Capability  Evaluation 

SCM 

Software  Configuration  Management 

SDD 

Software  Design  Document 

SDP 

Software  Development  Plan 

SDIO 

Space  Defense  Initiative  Office 

SDR 

System  Design  Review 

SEI 

Software  Engineering  Institute 

SEPG 

Software  Engineering  Process  Group 

SOW 

Statement  of  Work 

SPIP 

Software  Process  Improvement  Plan 

SPA 

Software  Process  Assessment 

SQA 

Software  Quality  Assurance 

SRR 

System  Requirements  Review 

SRS 

Software  Requirements  Specification 

SSA 

Source  Selection  Authority 

1  > 

SSAC 

Source  Selection  Advisory  Council 

SSDD 

’  System/Segment  Design  Document 

SSEB 

Source  Selection  Evaluation  Board 

SSET 

Source  Selection  Evaluation  Team 

SSEG 

Source  Selection  Evaluation  Guide 

SSP 

Source  Selection  Plan 

USA 

United  States  Army 

USAF 

United  States  Air  Force 

USN 

United  States  Navy 
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